Possible to pause dd, swap out destination to larger disk, and then continue?

Question

I'm copying a drive to a file, but underestimated the amount of space it would take up. So I am running out of space on the destination drive, and don't want to abort the process (it's been running quite a while on a damaged drive).

I can of course pause dd using CTRL-Z, and I'd like to swap out the destination drive for a larger one and copy the file over to the larger drive, then resume dd. Any ideas on how to accomplish this?

dd if=/dev/sdc conv=sync,noerror bs=64M | gzip -c -9 > /media/extD/drive.img.gz

EDIT: for those mentioning ddrescue as being a better tool to use, you're right. However, for whatever reason ddrescue kept barfing and erroring out on me, whilst dd has simply been chugging along without complaint.

Answer 1

It's possible, but kinda painful.

First, know that dd prints its current position in a copy (bytes copied) if sent the USR1 signal. so find the PID of dd, using either ps or something like pidof or pgrep (not POSIX and not on all unix-y systems IIRC).

A ps command that works for me (also using awk, in a debian environment):

ps aux|awk '/dd/ {print $2}'|grep -v awk

grep -v awk is necessary to prevent the PID of awk from printing as well.

Having the PID of dd, send the USR1 signal:

kill -USR1 [pid of dd]

the console window dd is running in will print how many bytes it's copied. You may now kill dd for real (ctrl+c, kill -9, whatever). I don't recall whether dd reports its progress on abort if killed this way in POSIX, so send the USR1 signal first.

dd may have now copied a few more bytes since you stopped it, so run:

head -c [number of bytes reported copied from dd] > \
     /path/to/drive/you/are/moving/to/filename.bin

to put a truncated copy on the destination disk. Instead of the precise number of bytes, you may want to choose something divisible by your desired block size, to speed up the transfer when you resume the copy. Just make note of whatever you choose, and make sure you are only truncating, not growing, the image.

Once you have this copied to the new drive, run:

dd if=/dev/sdc bs=64M skip=[truncated size divided by block size, e.g. 64000000] \
   of=/path/to/part2

If you have space for the remainder of the image on the smaller original disk, delete the non-truncated image you copied to the new disk in part 1 to free space, and have dd output it to that disk instead. When the transfer is complete, you can run

cat /path/to/part2 >> /path/to/part1

to add part 2 to the end of part 1, creating a complete disk image! Note that you will need at least as much free space on the disk part 1 is located on for all of part two to be appended to it.

If you don't mind doing the whole transfer over, I'd do cat /dev/sdc | gzip -c - > /path/to/imagefile.img.gz to create a gzip compressed archive. This can be written to a hard disk partition with something like zcat /path/to/imagefile.img.gz > /dev/sdX.

[copied from my comment into the answer]

additionally, I think (but do not remember for certain) that dd writes to stdout if of= is not specified. If this is true, you can skip writing part2 to a separate file and use:

dd bs=64M skip=[skip-block-count] if=/dev/sdc >> /path/to/part1

@MatijaNalis has rightly suggested using dd_rescue or ddrescue (two different programs that accomplish the same task) to copy the disk image. I'd do this if your partition/drive has erroneous sectors or other hardware faults.

Answer 2

You can't pause the process and resume it after moving the file on another drive¹. However you don't need to do that. Just kill the dd process, do your file transfer, and initiate a new copy process for the rest of the data.

To see where the first part stops, run gzip -l /media/extD/drive.img.gz. The “uncompressed” number is the number of bytes that were copied, and it's the offset where you need to start the new copy process.

To copy the rest of the data, tell dd to start at that offset NNNN.

dd if=/dev/sdc conv=sync,noerror,iflag=skip_bytes,skip=NNNN bs=64M |
gzip -c -9 >> /media/extE/drive.img.gz

I do not recommend using dd to copy data. Contrary to a common legend, there is no magic in dd that lets it access disks: the magic is all in /dev/sdc. Furthermore dd can lose data silently — I think that with the flags conv=sync,noerror it only replaces some data by zeros, so all the data that's successfully copied ends up at the right location, but do it at your own risk.

To copy a disk image from a working disk, just use cp or cat. To copy a part, tail -c +$((start_offset+1)) | head -c $bytes_to_copy.

To rescue data from a failing disk, use ddrescue. Ddrescue is smart about skipping unreadable data and keeps track of what it's successfully read. It makes multiple passes; with failing hard drives, letting the drive rest for a bit and trying again can often let you recover the whole thing, even if some sectors are unreadable at the first attempt.

ddrescue can use your existing partial copy and complete it, but you'll have to keep it uncompressed.

¹ _{Actually, you can, with ptrace — attach to the process with a debugger, pause it, update its internal data structures to point to the new file, resume. But even if you were familiar with how to do that, it would be more complicated than the straightforward solution with appending.}