dd backup of a raspberry pi over ssh with gzip compression

Question

I backup my raspberry pi "live" and remote from another linux computer, over ssh with gzip compression. I use this script:

#!/bin/bash
dt=`date +%y%m%d`
ssh pi@192.168.1.141 "sudo dd if=/dev/mmcblk0 bs=1M | gzip -" | dd of=./pibackup-$dt.gz

It works really well, but the resultant file is 26Gb, and that's too much (I think). I would like to backup only the content of the system and not the free clusters. I know dd make a whole backup, including free clusters too, but I thought that with the use of gzip I could reduce the size of the resultant file.

So looks out my system, when I type df -h:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G  3.1G   24G  12% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M   49M  415M  11% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   52M  201M  21% /boot
tmpfs            93M     0   93M   0% /run/user/1000
/dev/sda1       932G  184G  749G  20% /media/HDD

Answer 1

"Cold" backup is much safer than "live" backup.

On a running PI, run following command

cd $HOME; dd if=/dev/zero of=zero bs=8M; rm zero

It will write zeroes on all free space, which will give high compression rate. then put raspberry pi's microSD onto another system, for example, on a desktop running ubuntu and run

fdisk -l

I see microSD as /dev/sdb, then following command will compress the whole SD with :

gzip -c /dev/sdb > /backup/raspberry-pi.gz

To restore, run this command:

gzip -cd /backup/raspberry-pi.gz > /dev/sdb

It's recommended to take a full backup first before trying above procedure.

Answer 2

As suggested in the comments, you have a number of filesystem-level options for backing up the Pi. If you do this you will need to exclude the virtual filesystems /proc and /sys. You also should be aware that it will not include the non-filesystem boot image or your partition table; I'll provide a suggestion for backing that up separately at the end of this answer.

1. You can use rsync. This will give you a copy of the filesystem on your backup media rather than that a single image file.

   rsync -avzHP --exclude '/proc/' --exclude '/sys/' root@pi:/ /backup/pi.$(date +'%Y%m%d')
   

   If your local account isn't root you will probably want to include the --fake-super option for backup and restore. (It saves the remote ownership details although it can't actually apply them to the local backup.)

   If you do decide to go this route I'd strongly suggest you also look at rsnapshot to give you GFS backups without much extra disk space being used.

2. You can use tar, pax, or some other archiving tool. This will give you a compressed image

   ssh root@pi 'cd / && tar --exclude '/proc/' --exclude '/sys/' -czvf - .' > /path/to/backup.pi.$(date +'%Y%m%d').tgz
   

3. You can use other options such as duplicity. I've not used this seriously so I don't think I'm best qualified to give an example of how it would be used.

---

Having got the filesystem backup you also, presumably, want to be able to restore the minimal boot image.

On my Pi we have this structure (I've ignored partition three; you may have one, but it's probably been backed up as part of your filesystem):

fdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 29.7 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x42913321

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1         8192   137215   129024   63M  c W95 FAT32 (LBA)
/dev/mmcblk0p2       137216  8388607  8251392    4G 83 Linux

You need everything from the start of the disk to the start of partition 1 (/boot). In this particular display the sectors are in 512 byte blocks but we need to read in 4KiB blocks (SSD block size), so we divide all the numbers by eight:

# Copy the boot segment from the beginning of the disk
dd bs=4k count=$((8192/8)) if=/dev/mmcblk0 | gzip >img0.gz

You would restore this saved segment to the SSD card at /dev/mmbclk0 with a command like this. Note that it will irretrievably overwrite the destination device, so check it several times before using it, and do not blindly copy this example:

zcat img0.gz | dd bs=4k iflag=fullblock of=/dev/mmcblk0
partprobe /dev/mmcblk0

Answer 3

Here is my (trivial) dd-based image backup script for a Raspberry Pi SD card (to be run on another machine!):

#!/bin/sh
in=$1
out=$2
echo "Copying . . ."
time -p dd if=$in of=$out bs=4M status=progress
echo "Checking Source . . ."
set $(md5sum $in)
sum_in=$1
echo $sum_in
echo "Checking Destination . . ."
set $(md5sum $out)
sum_out=$1
echo $sum_out
if [ $sum_in = $sum_out ]
then
    echo "Compressing . . . "
    xz -6 --threads=2 --verbose $out && echo "DONE"
else
    echo "Checksum mismatch!"
fi

Save it as e.g. sd-verify-compress.

Make it executable:

$ chmod +x sd-verify-compress

Run it with (for example):

$ ./sd-verify-compress /dev/sdx bookworm64

Find what to use instead of /dev/sdx using lsblk:

$ lsblk -fmp

This is the file it creates:

-rw-r--r-- 1 pi pi 2.8G Nov 13 12:38 bookworm64.xz

My SD card is 32GB.