Script to monitor directory size change

Question

I need a bash script to monitor directory size change. The directory is a NFS file system with multiple users (500-600) who have sub-directories with vastly differing sizes. E.G /home/nfs/xxx/ccc etc. I need to be able to monitor, log and report for certain users that have directory sizes greater than e.g. 10GB or 20GB or indeed exceeded these sizes. This will need to be logged and the appropriate user sent a "friendly" email. I do not want to use utilities and/or tools to do this, rather with a bash script. This script would run on an ad-hoc basis.

Any help much appreciated.

MT

My efforts to date -

#!/bin/bash
set -x
DISK="/cluster/vvvvvvv1/nfs-home/zzz" # Verzeichnis
CURRENT=$(df -h | grep ${DISK} | awk {'print $4'}) # get disk usage from monitored disk
MAX="70%" # max nn% disk usage
DOMAIN="naz.ch"
Max Exceeded now find the largest offender
cd $DISK
for i in ls ; do du -sh $i ; done > /tmp/mail.1
sort -gk 1 /tmp/mail.1 | tail -1 | awk -F " " '{print $2}' > /tmp/mail.offender
OFFENDER=cat /tmp/mail.offender
echo $OFFENDER
du -sh $DISK/$OFFENDER > /tmp/mail.over70
mail -s "$HOSTNAME $DISK Alert!" "$OFFENDER@$DOMAIN"  < /tmp/mail.over70
check if current disk usage is greater than or equal to max usage.
if [ ${CURRENT} ]; then
  if [ ${CURRENT%?} -ge ${MAX%?} ]; then
    # if it is greater than or equal to max usage we call our max_exceeded function and send mail
    echo "Max usage (${MAX}) exceeded. The /home disk usage is it at ${CURRENT}. Sending email."
     max_exceeded
  fi
fi
init
main
#CLEANUP 

Answer 1

#assuming that your users are subfolders to same parent

disk=/home/disk # which contains users dirs user-{1..999}
limit='75%' 
current=$(df -k ${disk} | tail -1| awk '{print $5}')
max=10000000000 #in kilobytes (10G)

if [[ ${current//%/} -gt ${limit//%/} ]];then
    echo disk limit has been exceeded ${disk}
    # do your magick here
fi

find ${disk} -type f -exec du -k {} + | sort -nr | while read s f;do
    if test ${s} -gt ${max};then
        fsplt=(${f//\// }) # '/x/y/z' to 'x y z' as array
        echo "file size exceeded limits | user:${fsplt[2]} file:${f}";
        # do whatever here with file and user;
    else
        break; #cuz the list is sorted all remainning are smaller files
    fi
done

# you can customize find option to find only bigger than X size (optional)

Answer 2

There are some flaws with your script, some of them quite serious:

1. Never parse the output of ls. See Why not parse ls (and what to do instead)?

   It wasn't even necessary, your "for i in ls ; do du -sh $i ; done > /tmp/mail.1" doesn't need a loop, and would have been better written as:

     du -sch */ > /tmp/mail.1
   

   or even:

    du -sch */ | sort -gk 1 | tail -n 1 > /tmp/mail.1
   

2. You should almost always double-quote your variable expansions (including shell parameters like $1, $2, etc)...while there are some circumstances where you might not want to do that (e.g. you actually want the shell's usual word-splitting to occur), that is exceptionally rare and usually the exact opposite of what you want and need to do. In short: a good rule of thumb is "always quote unless you know exactly why you don't want to in this particular instance".

   See Why does my shell script choke on whitespace or other special characters?, $VAR vs ${VAR} and to quote or not to quote, Security implications of forgetting to quote a variable in bash/POSIX shells, and When is double-quoting necessary?.

   BTW, use double-quotes when you want variable interpolation to occur and single-quotes for fixed strings. Double-quotes will expand any variables etc inside them. Single-quotes won't. e.g. '$foo' is the literal string $foo, while "$foo" is the contents of a variable called $foo.

3. While it's not actually required, it is best practice to use all-lowercase or Mixed Case variable names for your own variables, just to ensure that they won't conflict with any environment variables used by the shell or other programs (which are almost always ALL-CAPS - e.g. the $HOSTNAME environment variable you're using - that's automatically sometime fairly early during the bootup process and exported to the environment so all child processes inherit it).

4. Since you're just trying to find the single "offender" using the most disk space (and how much space they're using), you don't even need to use a temporary file to do that. You could just do something like this:

   disk="/cluster/vvvvvvv1/nfs-home/zzz"
   subject="$HOSTNAME $disk Alert!"
   domain='naz.ch'
   read -r size offender < <(du -sch */ | sort -gk 1 | tail -n 1 |
                               awk -F'[[:space:]]+|/' '{print $1, $2}')
   if [ -n "$offender" ] ; then
     cat <<EOF | mail -s "$subject" "$offender@$domain"
       Your current disk usage is $size.  Please compress or delete
       any unnecessary files or directories.
   EOF
   fi
   

   But instead of doing this, you would be better off iterating over the output of du -sch, checking the size for each directory, and sending an email to the owner if it exceeds 10 or 20GB - the largest directory may be less than your 10-20GB limit. Note that a directory's name isn't necessarily the same as the owner's username - it's common for a user's home dir to have the same name as the user but it isn't always the case. Don't parse the output of ls to get a directory's owner - use stat instead.

   The above script fragment is just an example of how you could do this if you wanted to, it is certainly not a recommendation to actually do it this way.

---

BTW, you should probably be using the existing quota tools rather than writing your own. They not only have scripts to mail usage reports, they also allow you to set quota limits for each user and/or group which are enforced by the kernel. If you're using Linux, these will almost certainly be available as package for your distro - e.g. on Debian, they're in the quota package so you can install them with sudo apt-get install quota.

There's a pretty good basic tutorial to installing and using quota at Linux Quota - installation and configuration on Ubuntu and Debian - the installation method would be different for other distros (e.g. use yum or dnf on Fedora instead of apt) but configuration and usage would be the same. Arch Linux's wiki also has a good tutorial at Disk quota

Quota should probably be installed and configured on your NFS file server, rather than on your NFS client machine.