It seems that du discounts the #open files when calculating the disk usage, and this hit me when I saw that df is showing 100% usage, while du says only 40% is in use.
when i looked up on this, i came to realize that there are some open files that are used by the active processes that take up space
sudo /usr/sbin/lsof | grep services
Output:
java 3772 root 44r REG 9,5 76608 11649161 /services/apache-tomcat-6.0.20/lib/jsp-api.jar java 3772 root 74r REG 9,5 724225 16829492 /services/apache-tomcat-6.0.20/webapps/myapp/WEB-INF/lib/mysql-connector-java-5.1.10-bin.jar smbd 3794 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 3794 root 2w REG 9,5 1096 12320780 /services/samba/logs/smbd.log
the grand total of open files shown by this command (7th column in the output shown) is almost >60% of the total allotted 200GB space.
This is surprising to me that i can count on only 40% of my partition for actual storage. Really! is this normal.
I use this partition for hosting mysql database, tomcat server, samba, svn etc.
What are my options here? How can i tune this up to be able to use all 200GB for my data and not open files, if that is a normal expectation!
Newly added information
as per some of your suggestions here is the output from du
, df
and lsof
. the problem i want to resolve is not have too many open files which is unusually occupying more than 50% of the allotted disk space.
- My /services partition is originally allotted 184GB space (as shown in
df
command) - But my
du
command shows only 67GB (649GB-582GB) is occupied (excluding /services/BackupDir which is external drive) - while my
lsof
shows that there is 205GB open files (grand total of column#7), which is absurd because 205GB + 67GB is > 184GB!
#df showing the hard drive and the external drive on /dev/sde >df -h Filesystem Size Used Avail Use% Mounted on /dev/md0 43G 21G 21G 50% / /dev/md1 145M 37M 101M 27% /boot tmpfs 1.9G 0 1.9G 0% /dev/shm /dev/md3 230G 168G 63G 73% /sharefolder1 /dev/md4 230G 51G 179G 23% /sharefolder2 /dev/md5 184G 183G 1.4G 100% /services /dev/md6 138G 33G 106G 24% /home /dev/md7 94G 3.5G 90G 4% /users /dev/md8 230G 196G 23G 90% /sharefolder3 /dev/md9 230G 102G 116G 47% /sharefolder4 /dev/md10 459G 367G 69G 85% /sharefolder5 /dev/sde1 459G 141G 318G 31% /services/BackupDir/ext1 /dev/sde2 230G 85G 145G 37% /services/BackupDir/ext2 /dev/sde3 230G 102G 128G 45% /services/BackupDir/ext3 /dev/sde5 230G 47G 183G 21% /services/BackupDir/ext4 /dev/sde6 230G 172G 58G 75% /services/BackupDir/ext5 /dev/sde7 230G 38G 192G 17% /services/BackupDir/ext6 /dev/sde8 230G 188M 230G 1% /services/BackupDir/ext7#du showing only /services partition on the hard drive the subfolder BackupDir is the mounted external hard drive
sudo du --max-depth=1 -h /services
1.2G /services/svn 331M /services/httpd 582G /services/BackupDir 16K /services/lost+found 600M /services/netbeans-6.8 55G /services/apache-tomcat-6.0.20 9.3G /services/apache-tomcat-5.5.25 365M /services/mysql 3.7M /services/samba 649G /services
#lsof command on /services
sudo /usr/sbin/lsof | grep services
mysqld 3632 mysql cwd DIR 9,5 4096 15220739 /services/mysql/lib mysqld 3632 mysql 1u REG 9,5 51725 15220742 /services/mysql/log/mysqld.log mysqld 3632 mysql 2u REG 9,5 51725 15220742 /services/mysql/log/mysqld.log mysqld 3632 mysql 3uW REG 9,5 161480704 15220797 /services/mysql/lib/ibdata1 mysqld 3632 mysql 8uW REG 9,5 5242880 15220798 /services/mysql/lib/ib_logfile0 mysqld 3632 mysql 9uW REG 9,5 5242880 15220799 /services/mysql/lib/ib_logfile1 mysqld 3632 mysql 13u REG 9,5 1024 15220750 /services/mysql/lib/mysql/host.MYI mysqld 3632 mysql 14u REG 9,5 0 15220751 /services/mysql/lib/mysql/host.MYD mysqld 3632 mysql 15u REG 9,5 2048 15220753 /services/mysql/lib/mysql/user.MYI mysqld 3632 mysql 16u REG 9,5 2032 15220754 /services/mysql/lib/mysql/user.MYD mysqld 3632 mysql 17u REG 9,5 4096 15220747 /services/mysql/lib/mysql/db.MYI mysqld 3632 mysql 18u REG 9,5 17520 15220748 /services/mysql/lib/mysql/db.MYD mysqld 3632 mysql 19u REG 9,5 5120 15220759 /services/mysql/lib/mysql/tables_priv.MYI mysqld 3632 mysql 20u REG 9,5 3404 15220760 /services/mysql/lib/mysql/tables_priv.MYD mysqld 3632 mysql 21u REG 9,5 1024 15220762 /services/mysql/lib/mysql/columns_priv.MYI mysqld 3632 mysql 22u REG 9,5 0 15220763 /services/mysql/lib/mysql/columns_priv.MYD mysqld 3632 mysql 23u REG 9,5 1024 15220795 /services/mysql/lib/mysql/procs_priv.MYI mysqld 3632 mysql 24u REG 9,5 0 15220796 /services/mysql/lib/mysql/procs_priv.MYD mysqld 3632 mysql 26u REG 9,5 3072 15220792 /services/mysql/lib/mysql/proc.MYI mysqld 3632 mysql 27u REG 9,5 1852 15220793 /services/mysql/lib/mysql/proc.MYD smbd 3752 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 3752 root 2w REG 9,5 1096 12320780 /services/samba/logs/smbd.log smbd 3752 root 7u REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 3752 root 24w REG 9,5 1096 12320780 /services/samba/logs/smbd.log nmbd 3755 root 2w REG 9,5 2028 12320784 /services/samba/logs/nmbd.log nmbd 3755 root 3w REG 9,5 2028 12320784 /services/samba/logs/nmbd.log java 3772 root mem REG 9,5 17508 11649135 /services/apache-tomcat-6.0.20/bin/bootstrap.jar java 3772 root mem REG 9,5 9341 11649138 /services/apache-tomcat-6.0.20/bin/commons-daemon.jar java 3772 root mem REG 9,5 19399 11649146 /services/apache-tomcat-6.0.20/bin/tomcat-juli.jar java 3772 root mem REG 9,5 795412 11649645 /services/apache-tomcat-6.0.20/lib/poi-scratchpad-3.5-FINAL-20090928.jar java 3772 root mem REG 9,5 383040 11649637 /services/apache-tomcat-6.0.20/lib/commons-io-1.3.2-javadoc.jar java 3772 root mem REG 9,5 1516415 11649642 /services/apache-tomcat-6.0.20/lib/poi-3.5-FINAL-20090928.jar java 3772 root mem REG 9,5 1395266 11649159 /services/apache-tomcat-6.0.20/lib/jasper-jdt.jar java 3772 root mem REG 9,5 103201 11649158 /services/apache-tomcat-6.0.20/lib/jasper-el.jar java 3772 root mem REG 9,5 123820 11649154 /services/apache-tomcat-6.0.20/lib/catalina-ha.jar java 3772 root mem REG 9,5 88209 11649162 /services/apache-tomcat-6.0.20/lib/servlet-api.jar java 3772 root mem REG 9,5 724225 11650061 /services/apache-tomcat-6.0.20/lib/mysql-connector-java-5.1.10-bin.jar java 3772 root mem REG 9,5 1146570 11649156 /services/apache-tomcat-6.0.20/lib/catalina.jar java 3772 root mem REG 9,5 163177 11649641 /services/apache-tomcat-6.0.20/lib/commons-io-1.4-sources.jar java 3772 root mem REG 9,5 109043 11649639 /services/apache-tomcat-6.0.20/lib/commons-io-1.4.jar java 3772 root mem REG 9,5 197445 11649164 /services/apache-tomcat-6.0.20/lib/tomcat-dbcp.jar java 3772 root mem REG 9,5 69040 11649643 /services/apache-tomcat-6.0.20/lib/poi-contrib-3.5-FINAL-20090928.jar java 3772 root mem REG 9,5 27814 11649157 /services/apache-tomcat-6.0.20/lib/el-api.jar java 3772 root mem REG 9,5 391250 11649644 /services/apache-tomcat-6.0.20/lib/poi-ooxml-3.5-FINAL-20090928.jar java 3772 root mem REG 9,5 230149 11649155 /services/apache-tomcat-6.0.20/lib/catalina-tribes.jar java 3772 root mem REG 9,5 275779 11649634 /services/apache-tomcat-6.0.20/lib/commons-fileupload-1.2.1-javadoc.jar java 3772 root mem REG 9,5 513722 11649160 /services/apache-tomcat-6.0.20/lib/jasper.jar java 3772 root mem REG 9,5 87776 11649636 /services/apache-tomcat-6.0.20/lib/commons-io-1.3.2.jar java 3772 root mem REG 9,5 135544 11649638 /services/apache-tomcat-6.0.20/lib/commons-io-1.3.2-sources.jar java 3772 root mem REG 9,5 66144 11649165 /services/apache-tomcat-6.0.20/lib/tomcat-i18n-es.jar java 3772 root mem REG 9,5 499643 11649640 /services/apache-tomcat-6.0.20/lib/commons-io-1.4-javadoc.jar java 3772 root mem REG 9,5 49147 11649153 /services/apache-tomcat-6.0.20/lib/catalina-ant.jar java 3772 root mem REG 9,5 43742 11649166 /services/apache-tomcat-6.0.20/lib/tomcat-i18n-fr.jar java 3772 root mem REG 9,5 57779 11649632 /services/apache-tomcat-6.0.20/lib/commons-fileupload-1.2.1.jar java 3772 root mem REG 9,5 10807 11649152 /services/apache-tomcat-6.0.20/lib/annotations-api.jar java 3772 root mem REG 9,5 750470 11649163 /services/apache-tomcat-6.0.20/lib/tomcat-coyote.jar java 3772 root mem REG 9,5 73721 11649635 /services/apache-tomcat-6.0.20/lib/commons-fileupload-1.2.1-sources.jar java 3772 root mem REG 9,5 49021 11649167 /services/apache-tomcat-6.0.20/lib/tomcat-i18n-ja.jar java 3772 root mem REG 9,5 76608 11649161 /services/apache-tomcat-6.0.20/lib/jsp-api.jar java 3772 root mem REG 9,5 724225 16829492 /services/apache-tomcat-6.0.20/webapps/myapp/WEB-INF/lib/mysql-connector-java-5.1.10-bin.jar java 3772 root 1w REG 9,5 79186 9404433 /services/apache-tomcat-6.0.20/logs/catalina.out java 3772 root 2w REG 9,5 79186 9404433 /services/apache-tomcat-6.0.20/logs/catalina.out java 3772 root 5r REG 9,5 17508 11649135 /services/apache-tomcat-6.0.20/bin/bootstrap.jar java 3772 root 9r REG 9,5 9341 11649138 /services/apache-tomcat-6.0.20/bin/commons-daemon.jar java 3772 root 10r REG 9,5 19399 11649146 /services/apache-tomcat-6.0.20/bin/tomcat-juli.jar java 3772 root 11w REG 9,5 32125 9404434 /services/apache-tomcat-6.0.20/logs/catalina.2013-09-18.log java 3772 root 12w REG 9,5 24718 9404435 /services/apache-tomcat-6.0.20/logs/localhost.2013-09-18.log java 3772 root 13w REG 9,5 5336 9404436 /services/apache-tomcat-6.0.20/logs/manager.2013-09-18.log java 3772 root 14w REG 9,5 0 9404437 /services/apache-tomcat-6.0.20/logs/host-manager.2013-09-18.log java 3772 root 15r REG 9,5 795412 11649645 /services/apache-tomcat-6.0.20/lib/poi-scratchpad-3.5-FINAL-20090928.jar java 3772 root 16r REG 9,5 383040 11649637 /services/apache-tomcat-6.0.20/lib/commons-io-1.3.2-javadoc.jar java 3772 root 17r REG 9,5 1516415 11649642 /services/apache-tomcat-6.0.20/lib/poi-3.5-FINAL-20090928.jar java 3772 root 18r REG 9,5 1395266 11649159 /services/apache-tomcat-6.0.20/lib/jasper-jdt.jar java 3772 root 19r REG 9,5 103201 11649158 /services/apache-tomcat-6.0.20/lib/jasper-el.jar java 3772 root 20r REG 9,5 123820 11649154 /services/apache-tomcat-6.0.20/lib/catalina-ha.jar java 3772 root 21r REG 9,5 88209 11649162 /services/apache-tomcat-6.0.20/lib/servlet-api.jar java 3772 root 22r REG 9,5 724225 11650061 /services/apache-tomcat-6.0.20/lib/mysql-connector-java-5.1.10-bin.jar java 3772 root 23r REG 9,5 1146570 11649156 /services/apache-tomcat-6.0.20/lib/catalina.jar java 3772 root 24r REG 9,5 163177 11649641 /services/apache-tomcat-6.0.20/lib/commons-io-1.4-sources.jar java 3772 root 25r REG 9,5 109043 11649639 /services/apache-tomcat-6.0.20/lib/commons-io-1.4.jar java 3772 root 26r REG 9,5 197445 11649164 /services/apache-tomcat-6.0.20/lib/tomcat-dbcp.jar java 3772 root 27r REG 9,5 69040 11649643 /services/apache-tomcat-6.0.20/lib/poi-contrib-3.5-FINAL-20090928.jar java 3772 root 28r REG 9,5 27814 11649157 /services/apache-tomcat-6.0.20/lib/el-api.jar java 3772 root 29r REG 9,5 391250 11649644 /services/apache-tomcat-6.0.20/lib/poi-ooxml-3.5-FINAL-20090928.jar java 3772 root 30r REG 9,5 230149 11649155 /services/apache-tomcat-6.0.20/lib/catalina-tribes.jar java 3772 root 31r REG 9,5 275779 11649634 /services/apache-tomcat-6.0.20/lib/commons-fileupload-1.2.1-javadoc.jar java 3772 root 32r REG 9,5 513722 11649160 /services/apache-tomcat-6.0.20/lib/jasper.jar java 3772 root 33r REG 9,5 87776 11649636 /services/apache-tomcat-6.0.20/lib/commons-io-1.3.2.jar java 3772 root 34r REG 9,5 135544 11649638 /services/apache-tomcat-6.0.20/lib/commons-io-1.3.2-sources.jar java 3772 root 35r REG 9,5 66144 11649165 /services/apache-tomcat-6.0.20/lib/tomcat-i18n-es.jar java 3772 root 36r REG 9,5 499643 11649640 /services/apache-tomcat-6.0.20/lib/commons-io-1.4-javadoc.jar java 3772 root 37r REG 9,5 49147 11649153 /services/apache-tomcat-6.0.20/lib/catalina-ant.jar java 3772 root 38r REG 9,5 43742 11649166 /services/apache-tomcat-6.0.20/lib/tomcat-i18n-fr.jar java 3772 root 39r REG 9,5 57779 11649632 /services/apache-tomcat-6.0.20/lib/commons-fileupload-1.2.1.jar java 3772 root 40r REG 9,5 10807 11649152 /services/apache-tomcat-6.0.20/lib/annotations-api.jar java 3772 root 41r REG 9,5 750470 11649163 /services/apache-tomcat-6.0.20/lib/tomcat-coyote.jar java 3772 root 42r REG 9,5 73721 11649635 /services/apache-tomcat-6.0.20/lib/commons-fileupload-1.2.1-sources.jar java 3772 root 43r REG 9,5 49021 11649167 /services/apache-tomcat-6.0.20/lib/tomcat-i18n-ja.jar java 3772 root 44r REG 9,5 76608 11649161 /services/apache-tomcat-6.0.20/lib/jsp-api.jar java 3772 root 74r REG 9,5 724225 16829492 /services/apache-tomcat-6.0.20/webapps/myapp/WEB-INF/lib/mysql-connector-java-5.1.10-bin.jar smbd 3794 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 3794 root 2w REG 9,5 1096 12320780 /services/samba/logs/smbd.log smbd 3794 root 5w REG 9,5 1096 12320780 /services/samba/logs/smbd.log smbd 3794 root 7u REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb bash 10449 user1 cwd DIR 9,5 4096 14516225 /services/BackupDir smbd 10647 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 10647 root 2w REG 9,5 50351 12320785 /services/samba/logs/sridevi-pc.log smbd 10647 root 7u REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 10647 root 27w REG 9,5 50351 12320785 /services/samba/logs/sridevi-pc.log rsync 11030 user1 3r REG 9,10 21418176 23955858 /internalfile500GB/BackupDir/fileserver_services_bkup/internalfile-root-bkup/services/services/BackupDir/internalfile_repository_bkup/cng_repository/Deepak/DissertationDefense/Dissertation/Expression/Expression Encoder/Screen Capture Output/ScreenCapture_7-25-2011 4.55.33 PM.xesc smbd 11267 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 11267 root 2w REG 9,5 6298 12321110 /services/samba/logs/drwindows8.log smbd 11267 root 7u REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 11267 root 28w REG 9,5 6298 12321110 /services/samba/logs/drwindows8.log smbd 11334 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 11334 root 2w REG 9,5 5918 12321012 /services/samba/logs/nmo-dataprocess.log smbd 11334 root 7u REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 11334 root 32w REG 9,5 5918 12321012 /services/samba/logs/nmo-dataprocess.log smbd 11489 root mem REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 11489 root 2w REG 9,5 3484 12320915 /services/samba/logs/hippocampus-pc.log smbd 11489 root 7u REG 9,5 8192 12320776 /services/samba/etc/secrets.tdb smbd 11489 root 28w REG 9,5 3484 12320915 /services/samba/logs/hippocampus-pc.log sudo 11525 root cwd DIR 9,5 4096 14516225 /services/BackupDir grep 11526 user1 cwd DIR 9,5 4096 14516225 /services/BackupDir lsof 11527 root cwd DIR 9,5 4096 14516225 /services/BackupDir lsof 11528 root cwd DIR 9,5 4096 14516225 /services/BackupDir
#the size of deleted files in column#7 is 0 bytes!
sudo /usr/sbin/lsof | grep deleted
mysqld 3632 mysql 4u REG 9,0 0 7070979 /tmp/ibjpoC3F (deleted) mysqld 3632 mysql 5u REG 9,0 0 7070980 /tmp/ibTKRzCQ (deleted) mysqld 3632 mysql 6u REG 9,0 0 7070981 /tmp/ibvuExb1 (deleted) mysqld 3632 mysql 7u REG 9,0 0 7070983 /tmp/ibYLMOLb (deleted) mysqld 3632 mysql 11u REG 9,0 0 7070984 /tmp/ibQfuW1m (deleted) gpm 3690 root 0u CHR 136,0 0t0 2 /dev/pts/0 (deleted) gpm 3690 root 1u CHR 136,0 0t0 2 /dev/pts/0 (deleted) gpm 3690 root 2u CHR 136,0 0t0 2 /dev/pts/0 (deleted)
du -sh --apparent-size
too when runningdu
. Just an FYI. – slm Sep 20 '13 at 16:06du --appartent-size /services
is showing the same size as grand total of df available size for all the partitions. – user2105887 Sep 20 '13 at 16:36/dev/md5 192251868 190878620 1373248 100% /services
– user2105887 Sep 20 '13 at 16:38lsof
output. all log files are opened twice e.g., smbd.log, catalina.log, mysqld.log, etc. – user2105887 Sep 20 '13 at 16:44du
is missing could be shadowed by a mount, or kept in snapshots (if btrfs). Unlinked files is just one possibility. – Gabriel Sep 20 '13 at 16:45du
ordf
and thelsof
output you did show only covers a bit over 750k.du
does not miss open files unless they are also deleted. – psusi Sep 20 '13 at 17:36lsof
output is 205 MB, not GB. To havedu
not count files in the other mount points, use the -x switch, as indu -s -x /services
. – psusi Sep 21 '13 at 00:56