5

I'm using a raspberry pi running 24/7 and created a crontask that backups the complete sd card to my NAS on a regular basis. I checked such a backup by copying it back to an SD card and it worked perfectly. So far so good. Nevertheless I'm worried. Since such a backup takes hours over my network it seems obvious to me that it isn't a snapshot of one particular moment, but a mix of files from different moments since certain data on the card get modified while it's copying. For example, if there is a database installed or other running software changing regularly the contents of certain files. Or even changing system files. I can imagine that this might create inconsistencies in the backup copy, which in worst case, according to my perception, might create a corrupt backup that can't be used for a recovery. In worst case it might even not boot after recovery... Is this a real risk or do I misunderstand the way this dd command works?

Babavers
  • 59

3 Answers3

9

You have highlighted a real risk. If you are copying the underlying structure of a live filesystem you have no guarantee that what you have copied will be usable.

Even in the filesystem copy mounts successfully and looks good, you have no guarantee that the blocks you copied correspond to the blocks allocated for a particular file. (You might have a directory entry for a file that seems to be x MB in size, but only incomplete data inside its allocated blocks.)

This can be shown as an example where you are copying serially from the beginning of the SD card to the end. About mid-way through your backup, consider a directory entry gets created for a new file, and the file's contents written to the disk. The metadata might be captured because it's written to a directory physically located towards the end of the SD card, so the file seems to exist on the backup. However, the file contents could have been written to an earlier part of the disk that you've already processed, and so do not make it into your backup.

Don't do it.

The best option is to copy a snapshot (see LVM, snapshottable filesystems, and commercial software).

The next best is to perform a file-based backup (see rsync and rsnapshot); this is still subject to race conditions where two or more files can end up out of sync in the backup, and so not suitable for continuously changing files such as those of a database, but mostly most files will be copied successfully and correctly. If you need to backup a database in this scenario consider using a native database backup tool and then using rsync/rsnapshot to copy the resulting backup file. You will also need to copy separately and explicitly the disk partition table and non-filesystem partitions such as those involved in the boot process (dd can be your friend here, if you approach it cautiously).

Chris Davies
  • 116,213
  • 16
  • 160
  • 287
  • 2
    "... or perform a file-based backup (see rsync and rsnapshot)" - It's worth noting that those approaches are vulnerable to similar problems. I know rsync will complain if a file changes while it's being synced, but imagine a process updating two files (for example, a chunks file and an index file), and the data only makes sense if either both or neither of the files have been updated. But if rsync copies the first file before both are updated, and the second file after both are updated, the backup is inconsistent. Race conditions in backups are interesting :) – marcelm Jan 10 '21 at 10:55
  • 1
    @marcelm you are absolutely correct. I've updated my question to address this concern. Thank you for the poke – Chris Davies Jan 10 '21 at 13:06
4

You're correct in your assumptions: if you're simply taking a snapshot using dd, the resulting saved data might be inconsistent. Also, there can be errors while reading from the sdcard, as well as transmission errors (which is rare but possible).

You could try stopping all the writing daemons/applications, remount all the filesystems RO, then make a dd snapshot - that will guarantee data consistency but won't prevent read/transmission errors.

For maximum safety:

  1. Create a separate partition to backup data to.
  2. Backup twice and compare hashsums - they must match.
  3. Use par2 for backed up data and verify par2 volumes on creation.
Artem S. Tashkinov
  • 29,025
3

Yes, using dd to back up a mounted file system has definite risks. As someone else here said, "If you're lucky, you will discover the problems soon after the backup; if you are unlucky you will find them much later."

I don't think you are using the correct tool for the job. Since you have a Raspberry Pi (and if you are running RPi OS Buster), you may use image-utils. It uses rsync instead of dd, and so has very low risk of file corruption. It also creates an image file as backup - easily flashed to an SD card with rufus or etcher, and much smaller than a dd-created image file.

You have the option of updating an existing image backup, or creating a new one. In either case, backups are fast - a "full" backup takes about 15 minutes on my RPi.

image-utils is available for download here in the Raspberry Pi organization's forum. The author provides support in the same location, and it seems to be quite popular. I have no affiliation, but I do find it very useful.

EDIT @ 2021-06-07

This question has been declared a duplicate, and perhaps that's an accurate assessment from the general perspective of Linux & Unix. However, I wanted to call attention to the fact that there is (at least one) backup solution that works uniquely with the Raspberry Pi OS: image-utils.

Various questions re backups are asked in RPi SE from time to time, and for those that wind up here while searching for a solution, there is a more detailed answer available there.

Seamus
  • 2,925