16

If /var partition gets full on Production server then what's the solution ?

Below are my workaround:

  • If it is LVM Partition then we can extend it online.
  • We can compress logs.
  • We can remove old data.

Please suggest me more possible ways to solve and overcome this issue.

It would be helpful for me if you can share your experiences of this issue ever faced.

Gilles 'SO- stop being evil'
  • 829,060
Rahul Patil
  • 24,711

3 Answers3

11

My approach in any of these cases (something is filling up) is first to find the culprit.

I start by using du -sh *|grep G, continue with du -sh *|grep M. When I found it I start to research why that something starts filling up.

  • Do I need that high loglevel?
  • If logs - use logrotate (/etc/logrotate.d/) even for selv-made or custom programs
  • If this a real disk-hog I try to separate it into a LV of its own

In consequence a standard Linux-disk-layout for our servers looks currently like this:

  • /var LV with 2 GB
  • /var/log LV with 8 GB
  • /var/tmp LV with 4 GB

This is currently enough for almost any use-case we have.

Nils
  • 18,492
7

I once had a similar problem with a non-LVM partition that I solved by moving one of the directories to a more spacious partition and symlinking it back into place. In your case, for example, you can try:

mv /var/cache /more/spacious/partition/cache
ln -s /more/spacious/partition/cache /var/cache

Please note that I did this with a non-system directory and have so far observed no ill side effects. The case may be different with system directories, though. We need someone more knowledgeable to confirm/refute.

Edit

  • To be safer, you can do

    cp -a /var/cache /new/place/cache
rm -rf /var/cache
ln -s /new/place/cache /var/cache

    This ensures you won't lose your cache in case the mv call gets botched somehow (system crash, power outage,...)

  • To ensure that nothing is being written to the directory while you copy it, it's better if you do this via LiveCD.

Joseph R.
  • 39,549
  • I never tried this, but it might be good to do this with a Live CD, if you want to move system directories as @Joseph said. You should be careful, but it might just work ;) – Alko Aug 01 '13 at 13:53
  • @Alko you are correct, but we can check using lsof /var/cache is that used of not.. if not used that we can move to another location. – Rahul Patil Aug 01 '13 at 14:01
  • @RahulPatil and #Alko Good point. Adding to answer. – Joseph R. Aug 01 '13 at 14:04
0

The CIS Benchmarks require putting /home, /tmp and /var onto a separate partition to mitigate the worst outcome of a system freeze from root partition filling up.

There are other solutions to help it from filling up in the first place. Of course monitoring is one:
echo -e '#!/bin/bash\nCURRENT=$(df / | grep / | awk '\''{ print $5}'\'' | sed '\''s/%//g'\'') ; THRESHOLD=95; if [ "$CURRENT" -gt "$THRESHOLD" ] ; then mail -s "Disk Space Alert Used: $CURRENT" $EMAIL <<< $(hostname -i; uname -a); fi' >> /etc/cron.hourly/check-space && chmod +x /etc/cron.hourly/check-space

You can also set the system logs from filling up:
echo 'SystemMaxUse=200M' >> /etc/systemd/journald.conf && systemctl restart rsyslog

You can create a fixed size varfile and mount it to /var in /etc/fstab:
fallocate -l 600M /varfile && mkfs.ext4 /varfile echo -e "/home/varfile /home/user1 ext4 defaults,nofail 0 2" >> /etc/fstab (no dump, filecheck priority 1st is root)
Resize with this: fallocate -l 200M /tmpfile && mount /tmpfile /tmp && resize2fs /dev/loop1

I've safely done the file mounting with /home and /tmp dirs, but have not tested /var because I havent figured out how to copy the current contents onto the mountfile without using an attached livedisk or mounting the root filesystem. It seemed better for me to let the logs flux in free space and get monitoring which I need anyway, but I may revisit that for larger VMs than a minimal server.

Also, you can limit specific log sizes with a maxsize in individual conf files. However, things like /var/lib for apt lists and clamav definitions, as well as cache package installs in /var/cache/apt/archives, will grow aside from logs. The latter can be controlled through unattended-updates settings using apt clean periodically.

alchemy
  • 597