1

It seems that du discounts the #open files when calculating the disk usage, and this hit me when I saw that df is showing 100% usage, while du says only 40% is in use.

when i looked up on this, i came to realize that there are some open files that are used by the active processes that take up space

sudo /usr/sbin/lsof | grep services

Output:

java       3772      root   44r      REG                9,5     76608   11649161 /services/apache-tomcat-6.0.20/lib/jsp-api.jar
java       3772      root   74r      REG                9,5    724225   16829492 /services/apache-tomcat-6.0.20/webapps/myapp/WEB-INF/lib/mysql-connector-java-5.1.10-bin.jar
smbd       3794      root  mem       REG                9,5      8192   12320776 /services/samba/etc/secrets.tdb
smbd       3794      root    2w      REG                9,5      1096   12320780 /services/samba/logs/smbd.log

the grand total of open files shown by this command (7th column in the output shown) is almost >60% of the total allotted 200GB space.

This is surprising to me that i can count on only 40% of my partition for actual storage. Really! is this normal.

I use this partition for hosting mysql database, tomcat server, samba, svn etc.

What are my options here? How can i tune this up to be able to use all 200GB for my data and not open files, if that is a normal expectation!

Newly added information

as per some of your suggestions here is the output from du, df and lsof. the problem i want to resolve is not have too many open files which is unusually occupying more than 50% of the allotted disk space.

  1. My /services partition is originally allotted 184GB space (as shown in df command)
  2. But my du command shows only 67GB (649GB-582GB) is occupied (excluding /services/BackupDir which is external drive)
  3. while my lsof shows that there is 205GB open files (grand total of column#7), which is absurd because 205GB + 67GB is > 184GB!
#df showing the hard drive and the external drive on /dev/sde
>df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0               43G   21G   21G  50% /
/dev/md1              145M   37M  101M  27% /boot
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/md3              230G  168G   63G  73% /sharefolder1
/dev/md4              230G   51G  179G  23% /sharefolder2
/dev/md5              184G  183G  1.4G 100% /services
/dev/md6              138G   33G  106G  24% /home
/dev/md7               94G  3.5G   90G   4% /users
/dev/md8              230G  196G   23G  90% /sharefolder3
/dev/md9              230G  102G  116G  47% /sharefolder4
/dev/md10             459G  367G   69G  85% /sharefolder5
/dev/sde1             459G  141G  318G  31% /services/BackupDir/ext1
/dev/sde2             230G   85G  145G  37% /services/BackupDir/ext2
/dev/sde3             230G  102G  128G  45% /services/BackupDir/ext3
/dev/sde5             230G   47G  183G  21% /services/BackupDir/ext4
/dev/sde6             230G  172G   58G  75% /services/BackupDir/ext5
/dev/sde7             230G   38G  192G  17% /services/BackupDir/ext6
/dev/sde8             230G  188M  230G   1% /services/BackupDir/ext7

#du showing only /services partition on the hard drive the subfolder BackupDir is the mounted external hard drive

sudo du --max-depth=1 -h /services

1.2G /services/svn 331M /services/httpd 582G /services/BackupDir 16K /services/lost+found 600M /services/netbeans-6.8 55G /services/apache-tomcat-6.0.20 9.3G /services/apache-tomcat-5.5.25 365M /services/mysql 3.7M /services/samba 649G /services

#lsof command on /services

sudo /usr/sbin/lsof | grep services

mysqld 3632 mysql cwd DIR 9,5 4096 15220739 /services/mysql/lib mysqld 3632 mysql 1u REG 9,5 51725 15220742 /services/mysql/log/mysqld.log mysqld 3632 mysql 2u REG 9,5 51725 15220742 /services/mysql/log/mysqld.log mysqld 3632 mysql 3uW REG 9,5 161480704 15220797 /services/mysql/lib/ibdata1 mysqld 3632 mysql 8uW REG 9,5 5242880 15220798 /services/mysql/lib/ib_logfile0 mysqld 3632 mysql 9uW REG 9,5 5242880 15220799 /services/mysql/lib/ib_logfile1 mysqld 3632 mysql 13u REG 9,5 1024 15220750 /services/mysql/lib/mysql/host.MYI mysqld 3632 mysql 14u REG 9,5 0 15220751 /services/mysql/lib/mysql/host.MYD mysqld 3632 mysql 15u REG 9,5 2048 15220753 /services/mysql/lib/mysql/user.MYI mysqld 3632 mysql 16u REG 9,5 2032 15220754 /services/mysql/lib/mysql/user.MYD mysqld 3632 mysql 17u REG 9,5 4096 15220747 /services/mysql/lib/mysql/db.MYI mysqld 3632 mysql 18u REG 9,5 17520 15220748 /services/mysql/lib/mysql/db.MYD mysqld 3632 mysql 19u REG 9,5 5120 15220759 /services/mysql/lib/mysql/tables_priv.MYI mysqld 3632 mysql 20u REG 9,5 3404 15220760 /services/mysql/lib/mysql/tables_priv.MYD mysqld 3632 mysql 21u REG 9,5 1024 15220762 /services/mysql/lib/mysql/columns_priv.MYI mysqld 3632 mysql 22u REG 9,5 0 15220763 /services/mysql/lib/mysql/columns_priv.MYD mysqld 3632 mysql 23u REG 9,5 1024 15220795 /services/mysql/lib/mysql/procs_priv.MYI mysqld 3632 mysql 24u REG 9,5 0 15220796 /services/mysql/lib/mysql/procs_priv.MYD mysqld 3632 mysql 26u REG 9,5 3072 15220792 /services/mysql/lib/mysql/proc.MYI mysqld 3632 mysql 27u REG 9,5 1852 15220793 /services/mysql/lib/mysql/proc.MYD smbd 3752 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 3752 root 2w REG 9,5 1096 12320780 /services/samba/logs/smbd.log smbd 3752 root 7u REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 3752 root 24w REG 9,5 1096 12320780 /services/samba/logs/smbd.log nmbd 3755 root 2w REG 9,5 2028 12320784 /services/samba/logs/nmbd.log nmbd 3755 root 3w REG 9,5 2028 12320784 /services/samba/logs/nmbd.log java 3772 root mem REG 9,5 17508 11649135 /services/apache-tomcat-6.0.20/bin/bootstrap.jar java 3772 root mem REG 9,5 9341 11649138 /services/apache-tomcat-6.0.20/bin/commons-daemon.jar java 3772 root mem REG 9,5 19399 11649146 /services/apache-tomcat-6.0.20/bin/tomcat-juli.jar java 3772 root mem REG 9,5 795412 11649645 /services/apache-tomcat-6.0.20/lib/poi-scratchpad-3.5-FINAL-20090928.jar java 3772 root mem REG 9,5 383040 11649637 /services/apache-tomcat-6.0.20/lib/commons-io-1.3.2-javadoc.jar java 3772 root mem REG 9,5 1516415 11649642 /services/apache-tomcat-6.0.20/lib/poi-3.5-FINAL-20090928.jar java 3772 root mem REG 9,5 1395266 11649159 /services/apache-tomcat-6.0.20/lib/jasper-jdt.jar java 3772 root mem REG 9,5 103201 11649158 /services/apache-tomcat-6.0.20/lib/jasper-el.jar java 3772 root mem REG 9,5 123820 11649154 /services/apache-tomcat-6.0.20/lib/catalina-ha.jar java 3772 root mem REG 9,5 88209 11649162 /services/apache-tomcat-6.0.20/lib/servlet-api.jar java 3772 root mem REG 9,5 724225 11650061 /services/apache-tomcat-6.0.20/lib/mysql-connector-java-5.1.10-bin.jar java 3772 root mem REG 9,5 1146570 11649156 /services/apache-tomcat-6.0.20/lib/catalina.jar java 3772 root mem REG 9,5 163177 11649641 /services/apache-tomcat-6.0.20/lib/commons-io-1.4-sources.jar java 3772 root mem REG 9,5 109043 11649639 /services/apache-tomcat-6.0.20/lib/commons-io-1.4.jar java 3772 root mem REG 9,5 197445 11649164 /services/apache-tomcat-6.0.20/lib/tomcat-dbcp.jar java 3772 root mem REG 9,5 69040 11649643 /services/apache-tomcat-6.0.20/lib/poi-contrib-3.5-FINAL-20090928.jar java 3772 root mem REG 9,5 27814 11649157 /services/apache-tomcat-6.0.20/lib/el-api.jar java 3772 root mem REG 9,5 391250 11649644 /services/apache-tomcat-6.0.20/lib/poi-ooxml-3.5-FINAL-20090928.jar java 3772 root mem REG 9,5 230149 11649155 /services/apache-tomcat-6.0.20/lib/catalina-tribes.jar java 3772 root mem REG 9,5 275779 11649634 /services/apache-tomcat-6.0.20/lib/commons-fileupload-1.2.1-javadoc.jar java 3772 root mem REG 9,5 513722 11649160 /services/apache-tomcat-6.0.20/lib/jasper.jar java 3772 root mem REG 9,5 87776 11649636 /services/apache-tomcat-6.0.20/lib/commons-io-1.3.2.jar java 3772 root mem REG 9,5 135544 11649638 /services/apache-tomcat-6.0.20/lib/commons-io-1.3.2-sources.jar java 3772 root mem REG 9,5 66144 11649165 /services/apache-tomcat-6.0.20/lib/tomcat-i18n-es.jar java 3772 root mem REG 9,5 499643 11649640 /services/apache-tomcat-6.0.20/lib/commons-io-1.4-javadoc.jar java 3772 root mem REG 9,5 49147 11649153 /services/apache-tomcat-6.0.20/lib/catalina-ant.jar java 3772 root mem REG 9,5 43742 11649166 /services/apache-tomcat-6.0.20/lib/tomcat-i18n-fr.jar java 3772 root mem REG 9,5 57779 11649632 /services/apache-tomcat-6.0.20/lib/commons-fileupload-1.2.1.jar java 3772 root mem REG 9,5 10807 11649152 /services/apache-tomcat-6.0.20/lib/annotations-api.jar java 3772 root mem REG 9,5 750470 11649163 /services/apache-tomcat-6.0.20/lib/tomcat-coyote.jar java 3772 root mem REG 9,5 73721 11649635 /services/apache-tomcat-6.0.20/lib/commons-fileupload-1.2.1-sources.jar java 3772 root mem REG 9,5 49021 11649167 /services/apache-tomcat-6.0.20/lib/tomcat-i18n-ja.jar java 3772 root mem REG 9,5 76608 11649161 /services/apache-tomcat-6.0.20/lib/jsp-api.jar java 3772 root mem REG 9,5 724225 16829492 /services/apache-tomcat-6.0.20/webapps/myapp/WEB-INF/lib/mysql-connector-java-5.1.10-bin.jar java 3772 root 1w REG 9,5 79186 9404433 /services/apache-tomcat-6.0.20/logs/catalina.out java 3772 root 2w REG 9,5 79186 9404433 /services/apache-tomcat-6.0.20/logs/catalina.out java 3772 root 5r REG 9,5 17508 11649135 /services/apache-tomcat-6.0.20/bin/bootstrap.jar java 3772 root 9r REG 9,5 9341 11649138 /services/apache-tomcat-6.0.20/bin/commons-daemon.jar java 3772 root 10r REG 9,5 19399 11649146 /services/apache-tomcat-6.0.20/bin/tomcat-juli.jar java 3772 root 11w REG 9,5 32125 9404434 /services/apache-tomcat-6.0.20/logs/catalina.2013-09-18.log java 3772 root 12w REG 9,5 24718 9404435 /services/apache-tomcat-6.0.20/logs/localhost.2013-09-18.log java 3772 root 13w REG 9,5 5336 9404436 /services/apache-tomcat-6.0.20/logs/manager.2013-09-18.log java 3772 root 14w REG 9,5 0 9404437 /services/apache-tomcat-6.0.20/logs/host-manager.2013-09-18.log java 3772 root 15r REG 9,5 795412 11649645 /services/apache-tomcat-6.0.20/lib/poi-scratchpad-3.5-FINAL-20090928.jar java 3772 root 16r REG 9,5 383040 11649637 /services/apache-tomcat-6.0.20/lib/commons-io-1.3.2-javadoc.jar java 3772 root 17r REG 9,5 1516415 11649642 /services/apache-tomcat-6.0.20/lib/poi-3.5-FINAL-20090928.jar java 3772 root 18r REG 9,5 1395266 11649159 /services/apache-tomcat-6.0.20/lib/jasper-jdt.jar java 3772 root 19r REG 9,5 103201 11649158 /services/apache-tomcat-6.0.20/lib/jasper-el.jar java 3772 root 20r REG 9,5 123820 11649154 /services/apache-tomcat-6.0.20/lib/catalina-ha.jar java 3772 root 21r REG 9,5 88209 11649162 /services/apache-tomcat-6.0.20/lib/servlet-api.jar java 3772 root 22r REG 9,5 724225 11650061 /services/apache-tomcat-6.0.20/lib/mysql-connector-java-5.1.10-bin.jar java 3772 root 23r REG 9,5 1146570 11649156 /services/apache-tomcat-6.0.20/lib/catalina.jar java 3772 root 24r REG 9,5 163177 11649641 /services/apache-tomcat-6.0.20/lib/commons-io-1.4-sources.jar java 3772 root 25r REG 9,5 109043 11649639 /services/apache-tomcat-6.0.20/lib/commons-io-1.4.jar java 3772 root 26r REG 9,5 197445 11649164 /services/apache-tomcat-6.0.20/lib/tomcat-dbcp.jar java 3772 root 27r REG 9,5 69040 11649643 /services/apache-tomcat-6.0.20/lib/poi-contrib-3.5-FINAL-20090928.jar java 3772 root 28r REG 9,5 27814 11649157 /services/apache-tomcat-6.0.20/lib/el-api.jar java 3772 root 29r REG 9,5 391250 11649644 /services/apache-tomcat-6.0.20/lib/poi-ooxml-3.5-FINAL-20090928.jar java 3772 root 30r REG 9,5 230149 11649155 /services/apache-tomcat-6.0.20/lib/catalina-tribes.jar java 3772 root 31r REG 9,5 275779 11649634 /services/apache-tomcat-6.0.20/lib/commons-fileupload-1.2.1-javadoc.jar java 3772 root 32r REG 9,5 513722 11649160 /services/apache-tomcat-6.0.20/lib/jasper.jar java 3772 root 33r REG 9,5 87776 11649636 /services/apache-tomcat-6.0.20/lib/commons-io-1.3.2.jar java 3772 root 34r REG 9,5 135544 11649638 /services/apache-tomcat-6.0.20/lib/commons-io-1.3.2-sources.jar java 3772 root 35r REG 9,5 66144 11649165 /services/apache-tomcat-6.0.20/lib/tomcat-i18n-es.jar java 3772 root 36r REG 9,5 499643 11649640 /services/apache-tomcat-6.0.20/lib/commons-io-1.4-javadoc.jar java 3772 root 37r REG 9,5 49147 11649153 /services/apache-tomcat-6.0.20/lib/catalina-ant.jar java 3772 root 38r REG 9,5 43742 11649166 /services/apache-tomcat-6.0.20/lib/tomcat-i18n-fr.jar java 3772 root 39r REG 9,5 57779 11649632 /services/apache-tomcat-6.0.20/lib/commons-fileupload-1.2.1.jar java 3772 root 40r REG 9,5 10807 11649152 /services/apache-tomcat-6.0.20/lib/annotations-api.jar java 3772 root 41r REG 9,5 750470 11649163 /services/apache-tomcat-6.0.20/lib/tomcat-coyote.jar java 3772 root 42r REG 9,5 73721 11649635 /services/apache-tomcat-6.0.20/lib/commons-fileupload-1.2.1-sources.jar java 3772 root 43r REG 9,5 49021 11649167 /services/apache-tomcat-6.0.20/lib/tomcat-i18n-ja.jar java 3772 root 44r REG 9,5 76608 11649161 /services/apache-tomcat-6.0.20/lib/jsp-api.jar java 3772 root 74r REG 9,5 724225 16829492 /services/apache-tomcat-6.0.20/webapps/myapp/WEB-INF/lib/mysql-connector-java-5.1.10-bin.jar smbd 3794 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 3794 root 2w REG 9,5 1096 12320780 /services/samba/logs/smbd.log smbd 3794 root 5w REG 9,5 1096 12320780 /services/samba/logs/smbd.log smbd 3794 root 7u REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb bash 10449 user1 cwd DIR 9,5 4096 14516225 /services/BackupDir smbd 10647 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 10647 root 2w REG 9,5 50351 12320785 /services/samba/logs/sridevi-pc.log smbd 10647 root 7u REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 10647 root 27w REG 9,5 50351 12320785 /services/samba/logs/sridevi-pc.log rsync 11030 user1 3r REG 9,10 21418176 23955858 /internalfile500GB/BackupDir/fileserver_services_bkup/internalfile-root-bkup/services/services/BackupDir/internalfile_repository_bkup/cng_repository/Deepak/DissertationDefense/Dissertation/Expression/Expression Encoder/Screen Capture Output/ScreenCapture_7-25-2011 4.55.33 PM.xesc smbd 11267 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 11267 root 2w REG 9,5 6298 12321110 /services/samba/logs/drwindows8.log smbd 11267 root 7u REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 11267 root 28w REG 9,5 6298 12321110 /services/samba/logs/drwindows8.log smbd 11334 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 11334 root 2w REG 9,5 5918 12321012 /services/samba/logs/nmo-dataprocess.log smbd 11334 root 7u REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 11334 root 32w REG 9,5 5918 12321012 /services/samba/logs/nmo-dataprocess.log smbd 11489 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 11489 root 2w REG 9,5 3484 12320915 /services/samba/logs/hippocampus-pc.log smbd 11489 root 7u REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 11489 root 28w REG 9,5 3484 12320915 /services/samba/logs/hippocampus-pc.log sudo 11525 root cwd DIR 9,5 4096 14516225 /services/BackupDir grep 11526 user1 cwd DIR 9,5 4096 14516225 /services/BackupDir lsof 11527 root cwd DIR 9,5 4096 14516225 /services/BackupDir lsof 11528 root cwd DIR 9,5 4096 14516225 /services/BackupDir

#the size of deleted files in column#7 is 0 bytes!

sudo /usr/sbin/lsof | grep deleted

mysqld 3632 mysql 4u REG 9,0 0 7070979 /tmp/ibjpoC3F (deleted) mysqld 3632 mysql 5u REG 9,0 0 7070980 /tmp/ibTKRzCQ (deleted) mysqld 3632 mysql 6u REG 9,0 0 7070981 /tmp/ibvuExb1 (deleted) mysqld 3632 mysql 7u REG 9,0 0 7070983 /tmp/ibYLMOLb (deleted) mysqld 3632 mysql 11u REG 9,0 0 7070984 /tmp/ibQfuW1m (deleted) gpm 3690 root 0u CHR 136,0 0t0 2 /dev/pts/0 (deleted) gpm 3690 root 1u CHR 136,0 0t0 2 /dev/pts/0 (deleted) gpm 3690 root 2u CHR 136,0 0t0 2 /dev/pts/0 (deleted)

  • how do i respond to comments using full fromatting capabilities that i used to post my question? – user2105887 Sep 20 '13 at 15:40
  • 1
    I always use du -sh --apparent-size too when running du. Just an FYI. – slm Sep 20 '13 at 16:06
  • Thanks, but du --appartent-size /services is showing the same size as grand total of df available size for all the partitions. – user2105887 Sep 20 '13 at 16:36
  • but i can't make sense of what's going on here! look the used size (column 3) is more than the available size (column 4): /dev/md5 192251868 190878620 1373248 100% /services – user2105887 Sep 20 '13 at 16:38
  • also, i have noticed another strange thing in lsof output. all log files are opened twice e.g., smbd.log, catalina.log, mysqld.log, etc. – user2105887 Sep 20 '13 at 16:44
  • 2
    The data du is missing could be shadowed by a mount, or kept in snapshots (if btrfs). Unlinked files is just one possibility. – Gabriel Sep 20 '13 at 16:45
  • Can you please be more specific. I don't understand what do you mean by du is missing! Is there a command that you would suggest to check for unlinked files? – user2105887 Sep 20 '13 at 17:09
  • 2
    What are you talking about? You haven't shown the output of du or df and the lsof output you did show only covers a bit over 750k. du does not miss open files unless they are also deleted. – psusi Sep 20 '13 at 17:36
  • yes, i have posted only partial info. Can you please tell what's the best way to post my output to this thread, they are big and the comment can take only few lines... – user2105887 Sep 20 '13 at 17:43
  • @user2105887 You use the "edit" link below your post to edit the requested material into your question. – user Sep 20 '13 at 19:35
  • 1
    The total size of files in the lsof output is 205 MB, not GB. To have du not count files in the other mount points, use the -x switch, as in du -s -x /services. – psusi Sep 21 '13 at 00:56

3 Answers3

5

Having this discrepancy between du and df isn't uncommon, but having this large a discrepancy is - in my experience - very unusual. This type of discrepancy happens, in my experience, when an application file (usually a log file) is removed via rm, but is still open in the application, so the filehandle remains open and the kernel won't permit deletion of the actual data, but the inode entry in the directory is removed. Thus, df shows the space is still taken, but du which looks at an enumeration of files in a directory, doesn't see that file and shows that space available. The best suggestion I can give you is to work with the applications to try to reduce the number of filehandles they keep open.

John
  • 17,011
5

It seems that you've misread the output of the commands you ran several times.

the grand total of open files shown by this command (7th column in the output shown) is almost >60% of the total allotted 200GB space.

I have no idea where you got that figure. The total for the lines you show is about 800kB, which is about 0.0004% of 200GB.

If you added more lines than shown here, keep in mind that:

  • If a file was opened by multiple processes, or even on multiple descriptors by the same process (it happens), you've counted it multiple times.
  • Some of these files are on different filesystems.

How can i tune this up to be able to use all 200GB for my data and not open files, if that is a normal expectation!

There is nothing to tune up. You can use all your space. You're just making bizarre interpretations of the output of the commands you ran to measure disk usage.

sudo du --max-depth=1 -h /services

There are mount points under /services, so this sums up the size of files that are not on the /services filesystem but on /services/BackupDir/ext1 and its siblings.

The output from this command doesn't provide much useful information about the disk usage on /services. Pass the option -x to du to tell it not to descend into mount points.

sudo du -x -h /services

If the size reported by this command is less than the “occupied” size reported by df /services, there are two possible causes:

  • You have some files that are deleted but still open. These files still take up space, but they have no name so du won't find them. They would show up in the output of lsof. Run lsof +F1 /services to see a list of deleted but open files on /services.
  • There are files hidden behind some of the mount points under /services. Maybe one of your applications ran while these filesystems wasn't mounted as expected and therefore wrote files on the parent filesystem. When a filesystem is mounted on a directory, this hides the files in that directory, but of course the files are still there. Run the following commands to create an alternate view of /services without the lower mount points and explore that.

    mkdir /root/services-view
    mount --bind /services /root/services-view
    du /root/services-view/BackupDir/ext?
    
  • i can't believe it! like you said it is the mount point that was not created properly and the backup was written onto the disk. the df, du and lsof confused the whole thing. df is showing the size in KB, and i went with the same assumption with lsof, which was showing in bytes. – user2105887 Sep 23 '13 at 02:23
0

In addition to already suggested causes, it could be also following:

  • a different disk is mounted "over" the existing folder which is full of data
  • du will calculate the size spent of mounted disk and df will show really spent
  • solution: (when possible) unmount all non-root disks and check the size with du -md 1 again. Fix situation by moving hidden folder to some other place or mount on different place.