3

I have learnt about the dd cmd, this does the job perfect as one can create an image of a HD(with an OS already installed) and write it to other HD's.

The problem with this method is if you had a 2TB drive, and the OS+MBR/EFI was only 10GB, you would still have to copy the entire free space blocks of the 2TB drive, making the method very slow and inefficient. Upon writing this image to new HD's you would also have to write 2TB of data.

I am wondering if there is a more efficient way of doing this, Perhaps we can just copy the partition design meta data and then copy the OS files + MBR/EFI data? ignorning the freespace blocks?

Jimbo'sGun's
  • 41
  • What operating system? For Linux, you'd typically use an unattended installation system (PXEboot plus kickstart or autoyast). For something that's OS independent, can you buy a disk duplicator? – Mark Plotnick Jul 14 '16 at 23:29
  • @MarkPlotnick Altough a disk duplicator would work it still doesnt address the inefficiencies of copying empty space blocks. I am wondering of a way to get around this independantly of the OS used. – Jimbo'sGun's Jul 14 '16 at 23:32
  • Some disk duplicators have a "quick copy" mode. They understand disk partitioning and the structure of FAT, NTFS, ext2/3/4, and HFS filesystems, copying only the usable data. – Mark Plotnick Jul 14 '16 at 23:54

3 Answers3

3

One low-tech and fast possibility is to shrink the filesystem(s) to little more than what they contain, reproduce the partition table, and reproduce only the occupied part of each partition. After the copy, enlarge each filesystem to fill its partition.

But you should use tools that solve this problem for you. Partimage can save and restore a filesystem and only saves the used part. So can Clonezilla. So really the answer to your question is: use partimage or clonezilla.

Beware that when you do such deployment, you'll end up with identical filesystems on all machines. You'll have the same UUIDs for the filesystems, the same initial RNG state, the same private SSH key, etc. The UUID collisions don't matter as long as each clone evolves independently, but this violates the expectation that UUIDs are unique, and can be a problem e.g. if filesystem from multiple clones end mounted up on the same machine. The RNG state can be problematic if the instances are started up and used before they have time to be collected on their own. The private keys are definitely a problem. So after cloning, you should do a step of personalization on each instance (see also Moving linux install to a new computer).

An alternative approach is to create a new filesystem on each clone and copy the files individually (with cp -a — in my experience, GNU cp is good at preserving all metadata). However this is likely to be slower than a straight filesystem copy, especially on rotating media (HDD as opposed to flash) where reading the files will involve a lot of seeking back and forth.

Community
  • 1
Gilles 'SO- stop being evil'
  • 829,060
  • +1 for partimage and clonezilla. also worth pointing out that clonezilla actually uses partimage. and that cp (or rsync or any file-copy method) results in an optimally de-fragged target filesystem, rather than an exact clone of the current partition. file-copying is also essential if you want to clone your fs to an encrypted or compressed fs (or even just a different fs - e.g. from xfs to ext4). – cas Jul 15 '16 at 03:11
  • if anyone's wondering "why on earth would you want to move from xfs to ext4?"....well, ext4 has a very useful optimisation if the fs is on an SSD and you use fstrim. ext4 remembers which blocks have already been trimmed, so running fstrim will typically only take a minute or so. xfs doesn't, so will always trim ALL unused blocks (about half an hour on my 170GB / partition). – cas Jul 15 '16 at 03:16
  • @Gilles That seems the best approach. Once I have shrunk the OS partition and booted into linux live, I should only have to copy /dev/sda1(the MBR) and /dev/sda2(the OS shrunk) to my target external HD? If I have understood correct, wouldn't running dd if=/dev/sda1 of=/dev/sdc1 followed by dd if=/dev/sda2 of=/dev/sdc2 overwrite the first sda1 copy on sdc1 with the sda2 copy? – Jimbo'sGun's Jul 15 '16 at 15:50
  • @Jimbo'sGun's /dev/sda1 isn't the MBR, the MBR isn't part of any partition. If you aren't familiar with how partition tables are stored, stick to tools like partimage or clonezilla. – Gilles 'SO- stop being evil' Jul 15 '16 at 16:01
  • @Gilles I thought the smaller start of a HD is usually used to store the MBR(assumed sda1 being the first would contain the MBR within)... Guess somewhere along my studying I have misunderstood, as for using partimage/clonezilla I would still like to learn some fundamentals. In that case, would /dev/sda1 contain all that I need to copy?(ignoring /dev/sda2) – Jimbo'sGun's Jul 15 '16 at 16:10
  • @Jimbo'sGun's The MBR is before the first partition. But if you've already created the partitions on the target disk, you don't need to copy anything except the partitions. – Gilles 'SO- stop being evil' Jul 15 '16 at 16:17
1

Just an idea here Jimbo. What you can do is use.

cfdisk /dev/sda

changing drive letter as per your own requirements. Then just create a 10/11 Gb partition on it.

Then use dd against that newly created partition. Nesting partitions is possible. 

partprobe && dd if=/dev/sda1 of=/dev/sda1p2

Then create a GPT with cfdisk on the destination drive with cfdisk /dev/sda1 and copy data to the new destination with dd. 

dd if=/dev/sda1p2 of=/dev/sda1

One problem with dd is you will find it copy's byte for byte so it's slow and the UUID of the drives don't change. This can cause a clash if for instance you put a drive in one machine and try migrating it another machine that has the same UUID for LVM volumes. Also be very careful it's not called Disk Destroyer for nothing so make sure drives are correct.

As others mentioned kick starter, PXEboot or using a Rsync server and pulling down the filesystem after setting up partitions can also be done. If you are rolling out many systems but don't want to setup a server. Then consider something like an auto deployment script modified for your needs. Actually I have a very rough draft one you might be able to make use of here...

https://github.com/cyphernix/arch-enemy

Good Luck!

Keith Smith
  • 96
  • When you say change the drive letter and create a 10GB partition on it, is this basically to shrink the OS partition? When using dd against the newly created partition, wouldn't I have to also copy not only /dev/sda1(MBR) but also /dev/sda2(OS)? Also my machines are using MBR instead of GPT, I am not sure if some of your answer relies on using GPT - I would also like to remain with MBR method as changing this is likely to bring up new issues. – Jimbo'sGun's Jul 15 '16 at 15:57
  • No what I mean is the drive you are backing up to will be /dev/sda1 perhaps or /dev/sdb1, /dev/mmcblk1, /dev/vda etc... Although that varies from system to system depending on your own setup. df -h or just checking the output of dmesg will help in identifying the drive you need to back up towards. – Keith Smith Jul 19 '16 at 12:07
  • Also it should work just the same with MBR or GPT. /dev/sda1p1 for MBR and /dev/sda1p2 for OS. Nesting is possible to an extent. – Keith Smith Jul 19 '16 at 12:24
0

To deploy OS to multiple same machine, we should backup only operating system first(AOMEI Backupper is helpful). This is the best way due to time-saving and space-saving, since we all know that the smaller the file is the less backup time it costs and the less space it occupies.

kim
  • 1