11

I want to clone a hard disk using dd. Because I want to keep a process on the machine alive continuously, I would like to do this while the filesystem is still mounted. I understand this is not the "ideal" way to do this, but it also seems from Googling that it is possible.

The clone is being used as a backup; in the event of a hard disk failure I would like to have an image to dd back onto a new hard disk. The OS that is running lives on the disk I want to clone.

The process I have running does do some disk I/O but not with the disk I wish to clone. As far as I know, only the OS/system processes would be reading or writing to the disk while I do this operation. What I want to know is if this light use is likely to ruin the whole cloned image? I imagine that there's a danger of getting a few files corrupted if they're being written as they are read by dd, but I have no idea how likely it is to ruin the backup. Can anyone share some insights?

Short of putting the it on a disk and trying to start it, is there any way I can verify the integrity of the image?

Thanks!

alexandicity
  • 133
  • Since you say you want to use the spare disk "in the event of a hard disk failure", is there a reason you are choosing dd over a RAID 1 (mirroring) solution? RAID 1 system will even be able to keep running after a single HD failure. Linux/UNIX can create software RAIDs, even starting with existing non-RAID volumes. Replacing the disk requires a reboot, scheduled at any convenient time. (Hot swap hardware RAID is zero-downtime.) Leaving this as a comment, not an answer, as your question currently asks about the risks of using dd, not the best solution, which I of course respect. – type_outcast Nov 23 '15 at 19:55
  • 9
    RAID 1 is not a backup. – Rui F Ribeiro Nov 23 '15 at 19:57
  • 1
    If you want to dd from a mounted disk, you should probably remount it RO. This will allow it to stay mounted, but stop the OS writing to it, so you won't get an inconsistent copy. – Tom Hunt Nov 23 '15 at 20:37
  • Hi Tom, will the OS still be able to function if I mount it RO? Even if just for a limited time? – alexandicity Nov 26 '15 at 17:09
  • This is what happens when you sequentially capture parts of a view (filesystem) that can change in the meantime: panorama fail. Different parts of the result come from different states of the reality, they do not form a coherent picture. In order not to get an abomination, you need the view (filesystem) to be still (read-only) or you need to get it all in one shot (snapshot). – Kamil Maciorowski Dec 27 '23 at 13:40

3 Answers3

8

If you're lucky, the filesystem corruption will be detected as soon as you try to mount the copy. If you're unlucky, it won't be detected until later.

It's also possible that you'll manage to get a consistent copy of the filesystem except for the files that were modified during the copy. But I wouldn't count on it. It might work with ext4 as long as you don't create, delete or move any file, so that the directories aren't modified.

If you copy a filesystem that's mounted read-only, of course, everything will be fine. Except that you shouldn't use dd, use cat instead.

There are several reliable ways to clone a disk. Pick one of these, rather than one that practically guarantees corruption.

  • Some filesystems offer a clone functionality, for example btrfs. I don't think ext4 does.
  • If the filesystem is on Linux's native partition scheme, i.e. an LVM volume, you can make an LVM snapshot. That requires that you use LVM, rather than putting the filesystem directly on some other partition scheme such as MBR or GPT. You'll be left with a filesystem that wasn't cleanly unmounted, but represents a consistent snapshot of the original at a point in time.
  • If you can get the filesystem onto a RAID-1 array, you can clone it by adding a member to the array, waiting for it to synch, and detaching the new member. Here too you'll have a consistent but not clean snapshot. You can create a RAID-1 volume around an existing filesystem, but that requires an offline step to shrink the filesystem by 128kB.
  • You can make a file-level backup. That won't get you a consistent view of the filesystem, since copying files takes time, but it does guarantee at least that every file that wasn't modified during the backup will be backed up correctly.
Community
  • 1
Gilles 'SO- stop being evil'
  • 829,060
  • 2
    This should be at the top of man dd: If you're lucky, the filesystem corruption will be detected as soon as you try to mount the copy. If you're unlucky, it won't be detected until later. – Seamus Jan 14 '21 at 21:21
1

There is no harm in running a dd while running the server. The integrity of the backup depends mostly of what the server does, and what the partitions have there hosted.

Mostly static content has no problems at all, expect some problems with DB or log partitions. I would recommend to stop the DB servers on DB partitions.

If you are also backing up transaction-based filesystems, the backups should recover in no time.

dd could be used mostly to backup some server as it is, however you would need also file backups done at higher level.

dd will work quite well as a cloning alternative if the server is booted for an alternate medium and there is no activity at all.

As the previous post says, unless you have a specific need to used dd, there are better alternatives at there, free and commercial.

To name a few:

Clonezilla
Systemimager
dump / restore

Rui F Ribeiro
  • 56,709
  • 26
  • 150
  • 232
  • The preference to dd is because it's already installed, very simple to use and I can run it over SSH/netcat. Whether or not these are sufficient justification for its use over Clonezilla et al I'm not sure! I also assumed that Clonezilla and these other tools were largely dd or similar tool frontends, and so would suffer the same issues when reading from a live disk.. – alexandicity Nov 23 '15 at 19:53
  • Granted, I also often (ab)use netcat for dd and tars, mostly for tar. – Rui F Ribeiro Nov 23 '15 at 19:56
  • 1
    dd is likely to be way faster than those other tools you mention, though it depends on which of Clonezilla's options you choose, i suppose. also the thing you said about dd not honoring file blocking is kind of nonsense - dd honors input. if the input is a device file it will honor the device file's blocking to the extent the os requires (how could it not? block devices are blocked) and if it is a regular file it will honor whatever the file system provides it. zeroing empty space regularly on a disk being backed up is recommended, though. – mikeserv Nov 23 '15 at 23:58
0

The utility dd is a really basic tool. It operates at the raw bytes level of whatever you point it at. The upshot of this is that if something is writing to a file, you might copy the file during the time it's being written, likely resulting in a corrupted file, and certainly one that is simply wrong.

The things you would need to do to do this safely are already implemented in a variety of backup programs. It's simplest to use one of those.

Hack Saw
  • 1,024
  • 1
    dd is at block/raw level, it does not honor file blocking which is done at upper levels. – Rui F Ribeiro Nov 23 '15 at 19:37
  • What sort of programs are you thinking about there? Higher-level file backup tools? – alexandicity Nov 23 '15 at 19:57
  • Yep, or even lower level. You could make a simple mirroring RAID, or you could run rsync periodically, or use a GUI based backup tool. I use the first two. RAID for minimizing downtime, rsync offsite to get backups out of the building. – Hack Saw Nov 23 '15 at 20:59
  • Ok, thanks! Setting up RAID is a bit overkill here and would cause downtime while I set it up, but I can see that being a good long-term option. If I rsync everything (ie. / ), will that be sure to capture everything on the hard drive (assuming the hard drive is mounted as /)? Or might it skip some "non/quasi-file" things like the MBR or the dev files that might be needed? – alexandicity Nov 24 '15 at 13:12