0

How to clean logs or find largest files and reduce the space? also i do not understand 1.7T is not even used in this critical moment.

Mysqld is not running because 99% is used.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           20G   19G  279M  99% /
/dev/root        20G   19G  279M  99% /
devtmpfs         32G  268K   32G   1% /dev
/dev/md3        1.8T  196M  1.7T   1% /home
tmpfs            32G     0   32G   0% /dev/shm
/dev/root        20G   19G  279M  99% /var/named/chroot/etc/named
/dev/root        20G   19G  279M  99% /var/named/chroot/var/named
/dev/root        20G   19G  279M  99% /var/named/chroot/etc/named.conf
/dev/root        20G   19G  279M  99% /var/named/chroot/etc/named.rfc1912.zones
/dev/root        20G   19G  279M  99% /var/named/chroot/etc/rndc.key
/dev/root        20G   19G  279M  99% /var/named/chroot/usr/lib64/bind
/dev/root        20G   19G  279M  99% /var/named/chroot/etc/named.iscdlv.key
/dev/root        20G   19G  279M  99% /var/named/chroot/etc/named.root.key
Braiam
  • 35,991

3 Answers3

3

MySQL data is stored in /var/lib. There is no more space in /var. MySQL does not start. Pretty simple actually. Have a look at /var/log and clean it up. I would recommend something like (delete all gz files in /var/log) :

$ find /var/log -iname "*.gz" -delete

Of course, you might want to check what's being deleted first:

$ find /var/log -iname "*.gz"

1.7TB is given to /home. However /var is not /home. So /home is not being used. Simple as well.

You clearly should have separated /var on a much larger partition when you set up your system. Now, the best solution I can come across is to move the MySQL datadir to a larger partition (/home) :

$ mv /var/lib/mysql /home
$ ln -s /home/mysql /var/lib/mysql

If you don't want to use a link, you should reconfigure MySQL to use the new datadir stored in /home. Edit /etc/mysql/my.cnf and change this:

datadir = /var/lib/mysql

to this:

datadir = /home/mysql

Now that MySQL has space available, it should start without trouble. You should really reconsider your partition scheme if you're hosting such an important database though.

You might also want to investigate the contents of your logs, since applications shouldn't be writing so much to them if everything is going well. Something may be struggling to work properly somewhere.

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
John WH Smith
  • 15,880
2

This is a useful command to find the largest files:

du -ak /var | sort -nr | less

Usage:

du -ak summarizes the disk usage of all (-a) files in the /var partition and prints the size in kilobytes (-k).

sort -nr Concatenates the list of files and sorts them into a reversed (-r) numerical (-n) order.

less Will paginate the output so that you can see the largest offenders at the top and move back and forward through the list.

geedoubleya
  • 4,327
0

I would like to notice that it could also be a problem with mysql binary logs rather than with database files themself. Binlogs are placed in /var/log/mysql directory by default in many cases and they tend to consume few times more disk space than database files. It's because binlogs store all of the SQL data modifying queries (UPDATEs, INSERTs, etc.) to keep track of a database data changes. Binlogs should not be directly removed by shell rm command, just by mysql command on a running database engine. To do this mysql server has to be up. I think that moving /var/lib/mysql directory to the /home partition should release sufficient space on the / partition to make the mysql server able to start. So thus procedure should be as follows:

  1. Move mysql data to the /home partition as described by @John WH Smith.
  2. Change datadir variable as described by @John WH Smith and run the mysql server.
  3. Connect to the mysql server as the root user by a PMA, a mysql or some other client.
  4. Purge binary logs with the RESET MASTER or the PURGE BINARY LOGS TO commands (see the mysql documentation for details).
  5. Stop the mysql server. Edit the mysql config file. Remember value of the 'log_bin' variable. It contains a path and a name of binary log files.
  6. Move all the binlog files including a binlog index to the /home/mysql/ directory.
  7. Change the 'log_bin' variable value to 'log_bin = /home/mysql/mysql-bin.log' and then start mysql database.
Sebastian Piech
  • 99