12

When running dd command to copy an ISO on Linux, I get a single progress print that stays open for a long time (many minutes). Then another at the end.

The problem seems to be that a very large cache is being used which confuses dd 's output.

sudo dd bs=4M if=my.iso of=/dev/sdc status=progress

Output (first line shows for a long time).

1535115264 bytes (1.5 GB, 1.4 GiB) copied, 1.00065 s, 1.5 GB/s
403+1 records in
403+1 records out
1692844032 bytes (1.7 GB, 1.6 GiB) copied, 561.902 s, 3.0 MB/s

Is there a way to prevent this from happening so the progress output is meaningful?

ideasman42
  • 1,211
  • 1
    Why are you using dd for this? cat my.iso > /dev/sdc (as root, sudo -i) is simpler, les prone to errors and will probably be faster. – terdon Jul 21 '17 at 13:51
  • No use answering if OP already has seen the question. Can you give me the output of apt-cache policy coreutils – Hunter.S.Thompson Jul 21 '17 at 13:58
  • Dupe: https://askubuntu.com/questions/215505/how-do-you-monitor-the-progress-of-dd – quadruplebucky Jul 21 '17 at 14:15
  • @quadruplebucky - dont think this is a dupe, because I've read that and this question is about how to make one of the options work usefully. – ideasman42 Jul 21 '17 at 14:19
  • 2
    @quadruplebucky first of all, as ideasman42 explained, this isn't actually a dupe, but in any case, there's no such thing as cross-site duplicates so even if it were the same question, it's not something we can do anything about. – terdon Jul 21 '17 at 14:21

4 Answers4

18

From the first line we can tell dd has read and written 1.5GB in one second. Even an SSD can't write that fast.

What happened is that the /dev/sdc block device accepted it (writeback), but didn't send it to disk but buffered it and started writing to disk at the rate the disk can take it. Something like 3MiB/s.

The system can't buffer data indefinitely like that, there's only so much data it will accept to hold in that non-committed dirty state. So after a while (in your case, after more than 1.5GB have been written but less than 2 seconds have passed (as progress lines are written every second)), dd's write() system call will block until the data has been flushed to the disk (during which it cannot write progress messages). When it gets through, dd can send the few extra missing megabytes, and that happens within less than a second, so you get only one extra progress line.

To see a different behaviour, you could force the writes to be synchronous, that is not to return unless the data has been committed to disk. For instance by using oflag=sync or oflag=dsync or oflag=direct (not that I would advise doing that though).

Stéphane Chazelas
  • 544,893
  • 1
    Are you recommending against forcing synchronous writes altogether, or only against oflag=direct? – This isn't my real name Jul 21 '17 at 20:10
  • 2
    To expand on “not that I would advise doing that”: making the writes synchronous is a bad idea. It is likely to kill performance, and doing it often on an SSD will shorten its lifetime. Making one big write when the memory buffer gets full (not exactly what happens, but close enough) is much more efficient than many small writes. – Gilles 'SO- stop being evil' Jul 21 '17 at 23:00
9

When you write a file to a block device, use dd with oflag=direct. This uses O_DIRECT writes, which avoids using your RAM as a writeback cache. Note that to get good performance, oflag=direct usually needs a large block size.

This will avoid seeing impossibly fast progress, unless you have a weird device which itself has a very large RAM cache.

A lot of devices have a small amount of cache. In this case oflag=direct will show a realistic rate of progress. This is more meaningful, but it does not tell you everything you want to know :-). It does not guarantee the last writes are finished when dd says it is finished. You can make sure all writes are finished - and also check for any write errors - by using the option conv=fsync. This calls fsync() at the end to make sure the cache is flushed. Here is an example:

dd if=my.iso of=/dev/sdc oflag=direct bs=4M status=progress conv=fsync

Some people run the sync command afterwards, and understandably do not bother to remember conv=fsync. This is not quite as good. sync will not report whether one of the writes failed.

If the device has a very large RAM cache, you would have to use oflag=direct,sync. But normally I think of oflag=sync as a potential barrier to performance. You might want to increase the block size even further, to decrease the frequency of cache flushes. When doing very synchronous IO and reading multiple hardware blocks at a time like this, you might want to use double-buffering to maintain good performance, i.e. using a second dd command as in the link below.

Sometimes you might also want to pipe a disk image from another program, such as gunzip. In this case, good performance also depends on iflag=fullblock and piping through another dd command. There is a full example in the answer here: Why does a gunzip to dd pipeline slow down at the end?

sourcejedi
  • 50,249
1

Output flags oflag=nocache,sync can be used to ensure contents get written to drive with constant speed and constant buffer size instead of just being completely stuffed in system's cache and then written at the end:

dd if=/path/to/source.iso of=/path/to/usb-drive bs=4M status=progress oflag=nocache,sync

WARNING: do not use with low block size as it will lead to lower performance and very likely could lead to SSD health degradation. The default value is only 512 - very low. Value of bs=4M should be reasonable.

Demonstration - test with following commands:

# run in background in another terminal window to monitor write cache
while true; do cat /proc/meminfo | grep Dirty; sleep 0.1s; done;

run without flags (also, needs a manual sync to flush caches in my case)


dd if=/dev/random of=tmp.$(date '+%Y%m%d-%H%M%S') bs=1M count=128 status=progress
sync


run with oflag=nocache,sync


dd if=/dev/random of=tmp.$(date '+%Y%m%d-%H%M%S') bs=1M count=128 status=progress oflag=nocache,sync
kravemir
  • 4,174
-1

What Linux are you using ? If it's Debian based (Ubuntu, Mint, Debian etc..) I would use a program called Disks. It's a built in GUI disk based manager that can do the same as DD and give you consistent status.

Jagosix
  • 91
  • 2