1

I have a particular partition on my server (/var) that's filling up in a few days. Am running a busy mail server and all user mails go here.

However, now I need to increase the storage in order to accommodate more emails. 

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G  4.0K  3.9G   1% /dev
tmpfs           796M  556K  796M   1% /run
/dev/sda6       254G   32G  209G  14% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M  4.0K  5.0M   1% /run/lock
none            3.9G   80K  3.9G   1% /run/shm
none            100M     0  100M   0% /run/user
/dev/sda1       188M   66M  114M  37% /boot
/dev/sda7       1.6T  1.2T  364G  76% /var

So have been reading a lot of information online and there are several recommendations.

The first I have come across is using LVM partition. So I can add a second HDD to my server and then using LVM combine two hard-drives into one logical partition. While am still reading how to exactly do that, I read that if one drive fails, then the data is lost with no easy way of recovering. This has sort of sent me into looking for second alternative.

The second is using a union file system which enables me to merge several directories into one. I have so far come across these implementations; unionfs, aufs, mhddfs and overlayfs. In this case I might spin up a second HDD or borrow some space from less filled up partition like in my case root.

I have tried out overlayfs on my localhost because it's been added into the mainstream Linux kernel.

With lots of options, am not exactly sure what solution I should go with on a production critical mail server. I would appreciate some pointers. Thanks.

David Okwii
  • 445
  • 1
  • 4
  • 7

2 Answers2

7

First off, an overlay/union filesystem isn't the correct answer here. Those are for cases where you have a read-only filesystem that contains most of your data, and need to have some limited customization on top of that that is writable (for example, LiveCD's use overlay filesystems to allow the impression of a writable filesystem even though the media is read-only).

LVM is almost certainly what you want, and it doesn't have to be unreliable (it can do RAID setups with replication). Alternatively, you could just put in a new (bigger) hard drive, and put /var on that, although I would suggest just putting /var/mail or whatever your main mail storage directory is on that, and keeping the rest where it is.

In an ideal situation, you should be looking at getting multiple hard drives of the same size, and running a RAID10 or RAID5/6 set on those with /var/mail on top, and then working on getting your users to clean up old e-mail on the server (this situation is part of the reason that most mail providers put a cap on mail storage on the server).

heemayl
  • 56,300
Austin Hemmelgarn
  • 11,867
  • Hello thanks for your response. I forgot to mention that am using a cloud provider (softlayer) and they are providing SAN disks of up to 2TB in size. So that's a hard limit. – David Okwii Jul 25 '17 at 12:24
  • By bigger I just meant using one with more space than you have already. It will work fine with any size drive, but it doesn't make sense to use drives that give you no more space than you already have. – Austin Hemmelgarn Jul 25 '17 at 12:28
3

Your best option is to add at least two large-ish drives (e.g. 4TB or more) and use some form of RAID-1 or RAID-10 - if you don't have any storage redundancy in your "mission critical" service, you're doing it wrong.

If you add only two drives, use RAID-1. for four or more use RAID-10. I would advise against using RAID-5 or RAID-6 for mail because mail storage needs a LOT of I/O bandwidth, and RAID-5 is much slower than RAID-1/RAID-10 (and RAID-6 is even slower).

There are multiple ways of implementing RAID-1/RAID-10, including mdadm, and lvm. With either of these, you can create a RAID array, format it with your preferred filesystem, and mount it in place of /var/mail. You can also add more drives as needed in future, as long as your filesystem supports growing (e.g. xfs has xfs_growfs, ext2/3/4 has resize2fs).

Another option is to use btrfs. This has built-in support for RAID-like features, and automatically grows the filesystem when you add more storage drives to it. It also supports error-detection and correction, snapshots, sub-volumes, transparent file compression, and more.

ZFS has similar features to btrfs (and is what I personally use) but has the disadvantage that it's not (and will probably never be due to license incompatibility between CDDL and GPL) a part of the mainline kernel. It's only available as a patch or DKMS module. That's not a big deal these days, but it IS more work.

My recommendation would be to use either btrfs or zfs. IMO there's not really any good reason to use plain old mdadm or lvm these days unless you're already using it and have significant experience and investment in the technology.

There are numerous questions and answers on this site that detail how to set up mdadm, lvm, btrfs, and/or zfs and others.

Anyway, no matter how you implement your RAID or RAID-like filesystem, you'll have to transfer your old mail to the new filesystem with a minimum of downtime. In all cases, the procedure will be a lot like this:

  1. install the new drives. If you don't have hot-swap bays this will require some down-time.

  2. set up the new raid and/or filesystem

  3. mount it as /var/mail.new

  4. rsync /var/mail/ to /var/mail.new/

  5. You can repeat step 4 as often as you like until you have time to complete this procedure (probably best done out of business hours). Aside from some possible downtime in step 1, there should be no user-visible downtime up to this point....at most, they may have noticed the system is a bit slower than usual while the rsync jobs are running.

  6. stop your MTA (sendmail, exim, postfix, or whatever) and your pop/imap daemons, and anything else that writes to /var/mail/. If you have any users who login to a shell to read mail (e.g. with mutt), tell them to log out and don't let them login again until you've finished.

    In short, you need to stop anything and everything that writes to any file in /var/mail. You'll restart them later in step 13.

  7. run a final rsync from /var/mail/ to /var/mail.new/ to sync any changes since the last run.

  8. mv /var/mail /var/mail.old

  9. mkdir /var/mail

  10. unmount /var/mail.new and re-mount it as /var/mail (perhaps with mount --move /var/mail.new /var/mail as mentioned by Austin Hemmelgarn)

  11. examine the ownership and permissions of /var/mail.old and make sure that /var/mail has exactly the same.

    This is easily done with:

    chmod --reference=/var/mail.old /var/mail

    (note --reference requires GNU's version of chmod, which is standard on linux)

  12. edit /etc/fstab so that the new filesystem is always mounted as /var/mail after a reboot

  13. you can now restart your MTA, pop and imap services and allow users to login again. monitor it very closely to make sure it's working correctly.

  14. at your leisure, when you're sure that the new setup is working fine and you won't need the old mail directory any more, you can delete /var/mail.old and all its contents.

BTW, you'll end up with over 1TB free in /var after doing this. This may be excessive for your needs.

Community
  • 1
cas
  • 78,579
  • Thanks for the easy step-by-step guide on how to implement this. As I have mentioned in the comments on first answer, my cloud provider is giving me SAN disk of upto 2TB. Am finding out if they can provision 4TB HDDs. – David Okwii Jul 25 '17 at 12:26
  • 1
    FWIW, step 10 can also be done slightly more efficiently as mount --move /var/mail.new /var/mail. That will achieve the same thing with less typing (and without incrementing the mount count in the filesystem). – Austin Hemmelgarn Jul 25 '17 at 12:31
  • 1
    if you can only get one extra 2TB drive, it may be worthwhile partitioning the new drive (sdb, presumably) into a 1.6TB partition /dev/sdb1 (same size as sda7), and /dev/sdb2 with the remainder of the drive. move everything from /var except /var/mail to /dev/sdb2, then make a degreaded RAID-1 or btrfs with /dev/sdb1. copy /var/mail to it as above, then add /dev/sda7 to it as a raid-1 mirror drive when you've tested it all works fine. That will leave you ~400GB /var on /dev/sdb2 and ~1.6TB (plus compression - mail compresses extremely well - if using btrfs) as /var/mail. – cas Jul 25 '17 at 12:34
  • degraded, not degreaded. it's worth noting that this suggestion shares disk IOPS between /var/mail and both / and /var. It's no worse than what you have now (which already does that), but will not be as fast as having /var/mail on dedicated drives. it's a reasonable compromise between cost/hassle and maximising space. – cas Jul 25 '17 at 12:50
  • Finally, one thing I forgot to mention about ZFS - zfs makes it trivially easy to add a fast SSD and use that for caching writes (ZIL - ZFS Intent Log) and reads (L2ARC - Layer 2 Adaptive Read Cache). The biggest benefit of a ZIL comes from turning small random writes into what are effectively larger sequential writes, minimising head movement. Given that you rely on a cloud provider and their SAN, this probably isn't very relevant to you. – cas Jul 25 '17 at 12:53
  • Thanks @cas , Am definitely going to try out zfs and btrfs for myself, for now though am trying out the LVM/RAID-1 combo – David Okwii Jul 27 '17 at 08:45