20

I am trying to using the following dd command to clone remote xfs disk to another location.

ssh root@source_ip  "dd if=/dev/vda2 bs=16M conv=noerror,sync status=progress " | sudo dd of=/dev/nvme1n1 bs=16M conv=noerror,sync status=progress

However, I got the following error msg

828895133696 bytes (829 GB) copied, 39610.432258 s, 20.9 MB/s
1288490188800 bytes (1.3 TB) copied, 64529.849170 s, 20.0 MB/s
dd: error writing '/dev/nvme1n1': No space left on device

The source disk has 1TB and the target disk has 1.2TB.

Can anyone help to explain why can't perform the disk to disk clone in this situation? Thanks!

I want to try to recover the deleted files from the source disk, and I'm not sure whether DD is the only right tool for this situation.

muru
  • 72,889
Tim
  • 317
  • 1
    Please remove conv=sync from the right-hand dd and try the command line again. If that still gives you errors, please add iflag=fullblock to the right-hand dd. – Mark Plotnick Jun 10 '20 at 04:07
  • The different byte counts on first and second line (which come from the first and second dd, respectively) are a strong clue that, as Mark says, you need iflag=fullblock. – q.undertow Jun 10 '20 at 04:28
  • dd: bad argument: "iflag=fullblock" so this is not related to dd. I added an answer that is based on a standard dd implementation. – schily Jun 10 '20 at 07:32
  • 9
    dd is not a tool for cloning and will clone every single sector into the new disk even if it's unused. Use a proper cloning tool instead. They're smarter and much faster – phuclv Jun 10 '20 at 10:39
  • 3
    You're using Linux here based on your tags, I very strongly recommend that when cloning disks over the network, you don't use a network pipe, but use a rpoer network block storage protocol to expose the target disk on the source system or vice versa. NBD is what I tend to prefer for this, but ATAoE and iSCSI work too. This will let you use a proper cloning tool like ddrescue and turn most network reliability issues into performance issues instead of letting them corrupt data. – Austin Hemmelgarn Jun 10 '20 at 13:40
  • 2
    use xfsdump and xfsrestore since you are dealing with an XFS file system. Many online articles telling on how to do specifically what you are trying to do. – ron Jun 10 '20 at 15:04
  • Don't copy raw disk data from one disk to another (except if they are 100% identical and you know what you are doing). Instead copy files with tar or cpio. – Thorbjørn Ravn Andersen Jun 10 '20 at 23:48
  • As a general rule, using conv= options with dd is a mistake. – Mark Jun 11 '20 at 03:29
  • 3
    Incidentally, your title says you're cloning a disk. Your command shows you cloning a partition of a disk. Are you sure the target is (at least) the same size as the source? – Chris Davies Jun 11 '20 at 08:39
  • yeah you got fscked by the lack of iflag=fullblock - you need it on both dd's – hanshenrik Jun 11 '20 at 15:21
  • 1
    I want to try to recover the deleted files from the source disk, and I'm not sure whether DD is the only right tool for this situation. - no it's not. use ddrescue for that. https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Invoking-ddrescue – hanshenrik Jun 11 '20 at 15:23
  • Just wanted to say thanks for posting this question. I learned something about dd today, and a little about cloning disks too. Cheers! – DrDR Jun 16 '20 at 22:07

4 Answers4

31

Compare this answer (note bs=1K is used there):

dd is an old, cranky program initially intended to operate on tape devices. When you tell it to read one block of 1kB, it attempts to read one block. If the read returns less than 1024 bytes, tough, that's all you get.

conv=sync gets important when dd indeed reads less than it expected. In your case any time the pipe from ssh fails to deliver a full 16M block (because of bs=16M) to the local dd in one read, the conv=sync of the latter will fill the "missing" data with zeros (NUL bytes). But the real data is not missing. What the local dd considers missing will be delivered when it tries to read the next block.

In effect the local dd injects zeros in random(-ish) places of the stream. In case it's not obvious I emphasize: this corrupts the data, the resulting "clone" will be virtually useless.

Using conv=noerror,sync even on the remote (reading) side may be wrong. Compare this answer: What does dd conv=sync,noerror do? You need to really know how it works to use it properly.

For the local dd conv=noerror,sync you used makes little sense. Drop it or add iflag=fullblock (if supported). Frankly I'm not sure if conv=noerror,sync iflag=fullblock has any advantage over not using these options at all in a case when dd reads from a pipe.

Kamil Maciorowski
  • 21,864
  • 14
    Better still, don't use dd at all – Chris Davies Jun 10 '20 at 13:22
  • 3
    @roaima: what should be used, then? – a3nm Jun 10 '20 at 20:05
  • 9
    @a3nm cat should do, or pv for a nice progressbar. pv /dev/sda > /dev/sdb – gronostaj Jun 10 '20 at 20:31
  • 13
    Yes, in this example OK. But for more complicated use cases, dd has options (seek, skip, etc.) that you can't replicate with just cat – a3nm Jun 10 '20 at 20:42
  • 3
    @a3nm in those situations consider using dd. But generally for straightforward replication of disks or disk partitions, dd is most definitely not the way to go. (And that's what this question is about.) – Chris Davies Jun 10 '20 at 22:50
  • I want to try to recover the deleted files from the source disk, and I'm not sure whether DD is the only right tool for this situation. – Tim Jun 11 '20 at 00:28
  • @Tim it isn't by any means. – hobbs Jun 11 '20 at 01:52
  • 2
    @Tim for forensics, DD the origial disk to a file on the new disk. You can still use that file as a disk image. – Criggie Jun 11 '20 at 01:53
  • 2
    Whatever happened to --retry-partial? It basically fixed dd to do almost nothing stupid as it would use part reads to fixup part reads until the whole block was read or EOF was reached. – Joshua Jun 11 '20 at 17:04
23

As others have already identified, the problem is that you're using dd. Rule #1 of using dd is you do not use dd. Rule #2 of using dd is you do not use dd. Rule #3 of using dd (for experts only) is don't use the bs option.

There is no magic in dd. The magic is in /dev/*.

dd will happily mangle your data and setting a block size may or may not help performance. Only use dd if you've figured out how to use it safely and you've done benchmarks in your specific use case to determine block sizes that are faster that letting the system do its thing.

ssh root@source_ip  "cat /dev/vda2" | sudo tee /dev/nvme1n1 >/dev/null
Gilles 'SO- stop being evil'
  • 829,060
  • 2
    Probably worth ssh root@source_ip "gzip </dev/vda2" | zcat | sudo tee /dev/nvme1n1 >/dev/null for all those potentially empty blocks – Chris Davies Jun 10 '20 at 13:02
  • 5
    @roaima Yes, good point. Or simpler: ssh -C. – Gilles 'SO- stop being evil' Jun 10 '20 at 13:04
  • What advantage does tee provide here? – Binarus Aug 24 '23 at 08:20
  • 1
    @Binarus sudo tee myfile is one way to write myfile as root. sudo sh -c 'cat >myfile' is another. sudo cat >myfile wouldn't work since the surrounding shell would open myfile, and it's not running as root: only cat runs as root, but cat doesn't open its output, it just writes to its standard output. – Gilles 'SO- stop being evil' Aug 24 '23 at 09:30
  • Thank you very much, and +1. I have got your point. I didn't come to that idea because I'm mostly working as root in such cases (not normally). – Binarus Aug 28 '23 at 08:27
  • If somebody else comes here: With a Hitachi 80 GB SATA (Deskstar 7K250) disk from 2007 as destination, and with debian bullseye from 2023, I just ran into that "no space left" error regardless of what cloning method I used. That happened even without SSH being involved. I wanted to overwrite the disk and as root did cat /dev/random > /dev/sdc. That indeed overwrote the disk, but at the end lead to the "no space left" error; the same with dd if=/dev/random of=/dev/sdc .... With other disks, both methods work without that error. Maybe some disks simply report the wrong size to the O/S. – Binarus Aug 28 '23 at 08:38
8

For anything involving forensics, including undeleting files, dd is the RIGHT tool (at least for the read).

The problem, as others have pointed out, is that at the end of a ssh pipe, you aren't reading it blocked. So you either need iflag=fullblock on the second dd, or to just skip the second dd altogether.

flag=noerror is probably wrong. You want to know if it failed.

flag=sync is often a mistake. It tells dd to override the OS's buffering. You don't need that. At most, run sync afterwards. And for reading, it is meaningless.

Two different status=progress entries is a mistake. Both will report overwriting each other. You really just want to know how it finished. Omit status=...

bs=16M seems nice, but ideally bs should match the hardware block size. For classic disks, that was 512 bytes. More modern ones use 4k. I would recommend 4k, 16k, or 64k.

For disks that aren't full, compression is also good. The ssh -C solution is probably easiest.

So you could write:

ssh -C root@source_ip  "dd if=/dev/vda2 bs=64k" | sudo dd iflag=fullblock of=/dev/nvme1n1 bs=64k

or

sudo sh -c 'ssh -C root@source_ip  "dd if=/dev/vda2 bs=64k" >/dev/nvme1n1'

(this one might have ssh issues)
or

sudo chown `id -u` /dev/nvme1n1
ssh -C root@source_ip  "dd if=/dev/vda2 bs=64k" >/dev/nvme1n1
sudo chown root /dev/nvme1n1

or use the rather clever solution involving using tee.

testing

If you want to use the double dd form, it is probably good to test that you've got the right options before doing the full transfer. Add count=100 to the first dd so that it does a little bit of the disk. You should see both status lines in a few seconds and they should show the same amount transferred. If it works, repeat with the count=100 removed to do the full disk.

David G.
  • 1,369
3

The command dd was modeled after the command DDR (Disk Dump and Restore) from IBM mainframes from the 1960s and is mainly intended to convert and reblock block oriented I/O from/to block oriented disks. This is however not what you intend to do here.

The important problem in your use case is than you have an unreliable network pipe in the middle that does not grant to read specific block sizes.

If you are using a real dd program (not sure what you use since your output is incompatible to the standard) you should get output like this:

"%u+%u records in\n", <number of whole input blocks>,
    <number of partial input blocks>

"%u+%u records out\n", <number of whole output blocks>,
    <number of partial output blocks>

You need to use dd in a way that makes sure that you only get numbers > 0 before the plus sign and a 0 after the plus sign.

Since there is no grant to get a specific amount of data when reading from the ssh output pipe, you need to make sure to always try to read less than you would get. You could still use the following command to read from the source disk:

dd if=/dev/xxx bs=16x1024k

without problems, but you need to use this command to read from the pipe and to write to the destination disk:

dd ibs=1b obs=16x1024k of=/dev/yyy

Do not user other options (in special none of the non-standard option you are using). If the dd command that writes to the destination disk finally writes:

nnnn+0 records in
mmmm+0 records out

you have been successful and dd did never read less than 512 bytes from the input and thus could correctly reblock the output.

If you see partial blocks (> 0 to the right of the plus sign) for the input blocks, you need to repeat the copy process.

Note that because of the large output blocksize, you of course may get one partial block written to the destination dist at the end. So the summary message for the dd that writes to the destination disk may alternatively be:

nnnn+0 records in
mmmm+1 records out
schily
  • 19,173
  • 1
    If you have a block size that doesnt cleanly divide to the data size you can get a +1 at the end due to the last block being incomplete. – masterX244 Jun 10 '20 at 12:04
  • OK, but only for the output blocks. I'll correct that. – schily Jun 10 '20 at 12:15
  • I suspect dd is based on the DD statement in IBM mainframe's JES2 or JCL environment (early to mid 1960s). That actually specified the mapping between an internal filename and an external filename, and could also do certain actions on the file, like deleting it, or releasing a tape. It was not unusual for the work of a job to be in the DD statements, and the program to be IEFBR14 (a standard return immediately program). The DD statement's syntax tended to be: //<internal_name> DD =,... – David G. Jun 11 '20 at 01:40