161

I've not used dd all that much, but so far it's not failed me yet. Right now, I've had a dd going for over 12 hours - I'm writing an image back to the disk it came from - and I'm getting a little worried, as I was able to dd from the disk to the image in about 7 hours.

I'm running OSX 10.6.6 on a MacBook with a Core 2 Duo at 2.1ghz/core with 4gb RAM. I'm reading from a .dmg on a 7200rpm hard drive (the boot drive), and I'm writing to a 7200rpm drive connected over a SATA-to-USB connector. I left the blocksize at default, and the image is about 160gb.

EDIT: And, after 14 hours of pure stress, the dd worked perfectly after all. Next time, though, I'm going to run it through pv and track it with strace. Thanks to everyone for all your help.

Braiam
  • 35,991
eckza
  • 2,179
  • 7
    Not answering your question, but your times are quite high IMO. Did you remember to pass a bigger block size to dd other than the default 512 bytes? dd ... bs=16M is my suggestion, given your RAM, disk size and speed. – Juliano Apr 13 '11 at 20:29
  • I didn't, simply because I wanted to play it safe. I'll try that next time, though. Thanks. – eckza Apr 14 '11 at 01:26
  • In my experience, dd on Mac OS X has a tendency to freeze to the point where I can't even kill the process, but have to restart the system. I resort to doing work on a Linux VM then. – ssc Jun 12 '14 at 18:58
  • as pointed by @Juliano, a higher bs does make a huge difference in speed, by orders of magnitude – ccpizza May 13 '23 at 13:13

14 Answers14

188

You can send dd a certain signal using the kill command to make it output its current status. The signal is INFO on BSD systems (including OSX) and USR1 on Linux. In your case:

kill -INFO $PID

You can find the process id ($PID above) with the ps command; or see pgrep and pkill alternatives on mac os x for more convenient methods.

More simply, as AntoineG points out in his answer, you can type ctrl-T at the shell running dd to send it the INFO signal.

As an example on Linux, you could make all active dd processes output status like this:

pkill -USR1 -x dd

After outputting its status, dd will continue coping.

Caleb
  • 70,105
  • 10
    Oh, very cool. You can combine those with pkill -USR1 -x dd – Michael Mrozek Apr 13 '11 at 17:26
  • 1
    Checked the man page for kill, and there's no mention of -USR1 - what does that do? I'm really nervous about running any form of kill on this process, as I have a client waiting on this to finish. – eckza Apr 13 '11 at 18:06
  • 1
    Start another dd doing something easy and try it on that before you risk breaking your big one. I don't know what USR1 stands for but my guess is it's a custom (user 1?) signal that different process might handle differently. – Caleb Apr 13 '11 at 18:10
  • 2
    No, it definitely stopped the process and spit out "User defined signal 1". Good thing I tried it on something extraneous. :P – eckza Apr 13 '11 at 18:37
  • 3
    My Ubuntu box says to use $ kill -s INFO $pid; wait $pid. "On systems lacking the 'INFO' signal 'dd' responds to the 'USR1' signal instead, unless the 'POSIXLY_CORRECT' environment variable is set." – charlesbridge Apr 13 '11 at 18:44
  • 9
    @kivetros: On BSD systems, you need to send the INFO signal. Linux doesn't have a SIGINFO and uses USR1 instead. – Gilles 'SO- stop being evil' Apr 13 '11 at 18:50
  • 2
    @Gilles thanks for tracking down the different signal on BSD. – Caleb Apr 13 '11 at 19:58
  • 5
    The SIGUSRx signals are for programs to do what they want with, as opposed to having a standardized meaning. SIGWINCH, for example, is raised when the terminal has changed its size and the program might need to redraw its screen. The operating system doesn't send SIGUSRx's so they are available for custom uses. – LawrenceC Apr 14 '11 at 01:17
  • 11
    Sending dd the USR1 signal too soon after it has started (i.e. in a bash script, the line after you started it) will in fact terminate it. Put a 0.1 second sleep in between and it will output its progress properly. By the way, a very nice dd command to test USR1/INFO on is dd if=/dev/zero of=/dev/null. :) – Lauritz V. Thaulow Apr 14 '11 at 08:51
  • 11
    BTW, all "true" BSDs send SIGINFO to foreground process group if status character (Ctrl+T by default) is sent to terminal. But I don't know whether is it true for MacOSX. – Netch Nov 01 '12 at 20:14
  • 1
    I can confirm that OSX also sends SIGINFO when using ctrl-t. Great tip! – Dave Snigier Jan 04 '14 at 05:10
  • 4
    CTRL+T does it pretty well. – code ninja Dec 03 '14 at 19:24
  • 1
    @MichaelMrozek pkill -USR1 -x dd actually killed the dd process. – Dan Loewenherz Aug 03 '16 at 15:45
  • 1
    @DanLoewenherz Possibly you ran it too soon after starting dd? It should be functionally the same as using kill, it just saves the step of looking up the PID – Michael Mrozek Aug 03 '16 at 15:50
  • 1
    @MichaelMrozek Odd...nope, ran it about 5 mins after starting. Weird, maybe it was just a fluke? – Dan Loewenherz Aug 03 '16 at 15:59
  • 1
    @Dan it does work if you have the right signal for the right platfom. Play with dd if=/dev/zero of=/dev/null and see what you can get sending different signals. – Caleb Aug 03 '16 at 18:11
  • kill -USR1 on OSX just seems to kill the dd process, lamentably. – mwfearnley Jan 06 '17 at 14:56
  • When I used the command pkill -USR1 -x dd my dd process stopped, i did not get a progress status, and I had to restart it. Why would this have happened? – ob1 Jul 17 '17 at 17:23
  • 1
    @ob1 Sounds like you used the wrong signal for your platform. – Caleb Jul 18 '17 at 05:11
114

Under OS X (didn't try on Linux), you can simply type Ctrl+T in the terminal running dd. It will print the same output as kill -INFO $PID, plus the CPU usage:

load: 1.40  cmd: dd 34536 uninterruptible 3.49u 64.58s
5020305+0 records in
5020304+0 records out
2570395648 bytes transferred in 4284.349974 secs (599950 bytes/sec)

I found out about it reading this thread, and trying to open a new tab in my terminal but mixing +T with Ctrl+T.

Michael Mrozek
  • 93,103
  • 40
  • 240
  • 233
AntoineG
  • 1,263
27

For dd, you can send a signal. For other commands that are reading or writing to a file, you can watch their position in the file with lsof.

lsof -o -p1234    # where 1234 is the process ID of the command
lsof -o /path/to/file

If you plan in advance, pipe the data through pv.

17

As of coreutils v8.24, dd has native support for showing progress. Just add the option status=progress.

Example:

dd if=arch.iso of=/dev/sdb bs=4M status=progress

Source

techraf
  • 5,941
17

A more general way is to use iotop that displays the current amount of disk reading / writing per program.

EDIT: iotop -o show only programs which perform currently I/O operations (thanks Jason C for this comment).

jofel
  • 26,758
  • 1
    This is my preferred quick check method as well. iotop -o will hide processes that are not doing IO and make it easier to tell at a glance what's going on. – Jason C Jun 04 '14 at 19:51
13

I usually attach strace to such a running process (with the -p $PID option) to see if it stays blocked in a system call or if it is still active.

Or, if you feel nervous about sending a signal to the running dd, start another dd to validate if this works.

philfr
  • 1,088
  • 2
    How exactly would you go about attaching strace? Also, I did start another dd and send one of the suggested signals to it, and... it killed it. – eckza Apr 14 '11 at 13:15
  • 2
    If you know the pid of the running dd process, just do strace -p . You should see the log of all system calls called by the process (mostly read and write) – philfr Apr 14 '11 at 14:07
11

For next time, you can just use pv from the start (if it's available through your package manager, install it). This is a utility with the sole purpose of piping input to output and monitoring progress and speed.

Then, for writing an image to a drive, say with 4MB block size:

pv -ptearb /path/to/image.bin | dd iflag=fullblock of=/dev/whatever bs=4M

Aside from initial buffering (offset by a final sync, which can be done through dd if you want), this will show you a progress bar, average speed, current speed, and ETA.

The iflag=fullblock option forces dd to grab full blocks of input through pv, otherwise you're at the mercy of the pipe for block sizes.

To go the other way use dd to read and pv to write, although you have to explicitly specify the size if the source is a block device. For a 4GB device:

dd if=/dev/whatever bs=4M | pv -ptearb -s 4096m > /path/to/image.bin

You could also determine the size automatically, something like:

dd if=/dev/whatever bs=4M | pv -ptearb -s `blockdev --getsize64 /dev/whatever` > /path/to/image.bin

It really doesn't matter what order you do dd and pv in, it's entirely performance-related -- if the device you are reading to or from has optimal performance for certain blocksizes you want to use dd instead of pv to access that device. You can even stick a dd on both ends if you want, or not at all if you don't care:

pv -ptearb /path/to/image.bin > /dev/whatever
sync
Jason C
  • 1,383
  • 3
  • 14
  • 29
5

ddrescue will give you stats as it's running.

demo: http://www.youtube.com/watch?v=vqq9A01geeA#t=144s

  • 4
    This might be helpful for the next time, but it won't help the OP to understand if the current command is frozen or not. –  Nov 02 '12 at 13:11
4

Sometimes you may not be able to use the INFO or USR1 signal because the stderr stream of the dd process is not accessible (e.g. because the terminal in which it was executed was already closed). In this case, a workaround is to do the following (tested on FreeBSD, may be slightly different on Linux):

  1. Use iostat to estimate the average write rate (MB/s) to the target device, e.g.:

    iostat -d -w30 ada0

    Substitute your target device name for ada0 here, and wait a minute for it to give a couple results. The "w" parameter determines how many seconds between samples. Increasing it will give a better average estimate with less variance, but you'll have to wait longer.

  2. Use ps to determine how long dd has been running:

    ps -xo etime,command | grep dd

    Convert this to seconds to get total seconds of runtime.

  3. Multiply total seconds of runtime by average write rate to get total transferred MB.
  4. Get the device size in MB with:

    grep ada0 /var/run/dmesg.boot

    Substitute your target device name for ada0. Divide the result by the average write rate to get the total transfer time in seconds. Subtract the time it's been running so far to get time remaining.

This strategy only works if dd has been writing continuously at the current average write rate since it began. If other processes are competing for the CPU or I/O resources (including the I/O bus) then it may reduce the transfer rate.

4

I started using dcfldd(1), which shows dd operations in a better way.

Kartik M
  • 331
3

While dd is executing, run this in another terminal as root:

while pgrep ^dd; do pkill -INFO dd; sleep 1; done

It prints the dd status every 1 second in the original terminal window where dd is executing, and quits when the command is done.

ccpizza
  • 1,723
3

You can use progress which, in particular, shows the progress of a running dd. It uses /proc/$pid/fd and /proc/$pid/fdinfo which you can also monitor by hand.

Stephen Kitt
  • 434,908
jofel
  • 26,758
1

The wchar line (written characters) in /proc/$pid/io can give you precise information about the dd process. As long as it changes, your dd is still working!

Here is a neat little php script, which you can save and then execute with php filename.php during the dd to display the written bytes. The nice benefit of watching /proc/$pid/io over kill -USR1 $(pidof dd) is that you do not have to switch between terminals, which is not always an option.

<?php

/** Time between refreshs in seconds */
$refresh = 1;


/**
 * Start of Script 
 */

if (!($pid = exec('pidof dd')))
    exit("no dd running\n");

$history = array();
$break_ms = $refresh * 1000000;
$start_time = exec("ls -ld /proc/$pid --time-style=+\"%s\" | egrep -o [0-9]{10}");


fprintf(STDOUT, "PID: %s\n", $pid);
fprintf(STDOUT, "START TIME: %s\n\n", date("Y-m-d H:i:s", $start_time));


while (true) {
    if (isset($curr))
        array_push($history, $curr);

    if (count($history) > 10) array_shift($history);
    $oldest = reset($history);
    $latest = end($history);

    /**
     * get number of written bytes from /proc/$pid/io
     */
    #if (!($curr = exec("cat /proc/$pid/io | grep ^write_bytes | sed 's/write_bytes: //g'")))
    #    break;

    /* prepare proc_open() parameter */
    $descriptorspec = array(
        0 => array('pipe', 'r'), // stdin
        1 => array('pipe', 'w'), // stdout
        2 => array('pipe', 'w'), // stderr
    );

    $process = proc_open("cat /proc/$pid/io | grep ^write_bytes | sed 's/write_bytes: //g'", $descriptorspec, $pipes);
    if (!is_resource($process)) break;

    $stdout = stream_get_contents($pipes[1]);
    $stderr = stream_get_contents($pipes[2]);
    proc_close($process);

    if (!empty($stderr)) break;
    $curr = trim($stdout);

    /**
     * caculate elapsed time from start */
    $time_elapsed = time() - $start_time;

    /**
     * avg speed since start */
    $avg = $time_elapsed > 0 ? round($curr / $time_elapsed) : 0;

    /**
     * avg speed of last 10 updates */
    if (count($history) > 0)
        $speed = human_file_size(round(($latest - $oldest) / count($history) / $refresh));

    $output = sprintf("\rBYTES WRITTEN: %s [%s]  ::  CURRENT: %s/s  ::  AVERAGE: %s/s  ::  ELAPSED: %s", $curr, human_file_size($curr), isset($speed) ? $speed : 0, human_file_size($avg), gmdate("H:i:s", $time_elapsed));
    printf("%s%s", $output, str_repeat(" ", exec("tput cols") - strlen($output)));

    usleep($break_ms);
}

fprintf(STDOUT, "\ndd has finished!\n\n");

function human_file_size($size,$unit="") {
  if( (!$unit && $size >= 1<<30) || $unit == "GB")
    return number_format($size/(1<<30),2)." GB";
  if( (!$unit && $size >= 1<<20) || $unit == "MB")
    return number_format($size/(1<<20),2)." MB";
  if( (!$unit && $size >= 1<<10) || $unit == "kB")
    return number_format($size/(1<<10),2)." kB";
  return number_format($size)." bytes";
}
1

If you write to a slow drive, for example a USB drive, you may want to know not only the progress of the command dd itself, but also the progress of actual writing to the target device.

  • One way way know when the process has finished is to run sync after the dd command and wait for it to finish flushing the buffers, so that the terminal window returns to prompt. But there is still no display of progress. You can monitor the progress of flushing by watching the 'dirty' data for example with the shellscript watch-flush, that is part of mkusb. This shell-script uses data from the system file /proc/meminfo.

  • It is more straightforward to modify the dd command line to make it flush the buffers regularly, for example after writing each mibibyte, and at the same time show the progress, for example

    sudo dd if=file.img bs=1M of=/dev/sdx status=progress oflag=dsync
    

    Please check and double-check, that you specify the correct target device. Otherwise you might overwrite valuable data. dd does what you tell it to do without any questions, and for this reason it has earned the nickname 'Data Destroyer'.

  • In Ubuntu and Debian you can also use mkusb to perform the cloning task. It will wrap a safety belt around dd: help you identify the target device and let you double-check it before you launch the process.

sudodus
  • 6,421