1

I am trying to understand why the backup process in one of the servers is taking so long and not copying the data correctly.

This is the size of the external HDD partition. 

df -h /dev/sdb1
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1             985G  362G  573G  39% /media/backup

The folder that I am trying to copy is lesser than the size of the external HDD partition. 

df -h /dev/sda4
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda4             3.4T  867G  2.4T  27% /mounts

Everything just seemed fine. I started the backup process almost a week back and still it has not completed. I realized it might be an issue with the USB port speed but I was wrong.

I also suspected it might be a corrupted file system that might be causing the error. I wanted to run fsck on the /dev/sda4 file system. But when I checked some of the script output, I read some error messages as, 

rsync: mkstemp "/media/backup/2014-06-18_09-36/mounts/ no space left on device (28)

From here, I read that 5% of the disk space would be allocated for root file system. I do not understand if that is related to my problem. Also, from here, I read the inode size cannot be greater than 2 GB. I believe it should be something that I investigate. But am not sure if that is the problem though. I would appreciate more pointers towards the right direction.

EDIT:

The output of df -i command is as below. 

df -i /dev/sdb1
/dev/sdb1            65544192 65279823  264369  100% /media/backup
df -i /dev/sda4
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda4            229957632 171244050 58713582   75% /mounts
Community
  • 1
Ramesh
  • 39,297
  • What about df -i? Also, what filesystem is used on the external drive? – derobert Jun 24 '14 at 16:02
  • @derobert, please see the edit to the question with the df -i command output. It seems promising. Does it mean something? – Ramesh Jun 24 '14 at 16:05
  • I'm not sure what rsync uses mkstemp() for, but to guess, probably so that it doesn't create partial files on error. Running out of inodes does seem like a likely cause. – goldilocks Jun 24 '14 at 16:06
  • Ain't some kind of quota applied on that device? – psimon Jun 24 '14 at 16:06
  • @goldilocks, thanks. I also suspect the same. – Ramesh Jun 24 '14 at 16:06
  • @psimon, I don't think so. There is no quota in this file system. – Ramesh Jun 24 '14 at 16:08
  • Then @derobert 's answer is right, you've run out of indodes. – psimon Jun 24 '14 at 16:10
  • @Ramesh BTW: You are getting close to running out of inodes on the source volume too; note how your inode usage % is much higher than your space usage. I suggest you begin planning to reformat that as well. – derobert Jun 24 '14 at 16:13
  • @derobert, this system is going to be reinstalled with newer RHEL6. So I really do not worry about the source system as it is no longer used. I just want to backup the data and then reinstall the OS. :) – Ramesh Jun 24 '14 at 16:14
  • @Ramesh when you reinstall the OS, make sure to use a lower inode_ratio (i.e., more inodes). You need a ratio appropriate for the data you're storing, which appears to be a lot of small files. – derobert Jun 24 '14 at 16:15
  • @derobert, sure. I will do that. Thanks again :) – Ramesh Jun 24 '14 at 16:16

1 Answers1

5

You have run out of inodes on your backup drive. That's the out of space error you're seeing.

Each file (basically) takes one inode. Unfortunately, with most filesystems there isn't a way to add more inodes, except mkfs.

Example: with ext4, you pick the number of inodes created (at mkfs time) directly with the -N option, or as a ratio to the volume size with -i. The various usage types (-T mainly vary the inode ratio).

derobert
  • 109,670
  • thanks. So if I just copy folder by folder will that help or I will still get the same error? – Ramesh Jun 24 '14 at 16:07
  • @Ramesh No. You have more inodes used on your source filesystem than total on your destination. Fundamentally, you can't copy that source to that destination. It simply doesn't fit. – derobert Jun 24 '14 at 16:09
  • got it. So, I believe I should put the files just in different partitions to accomodate them. Though I have full space available, I cannot use them. – Ramesh Jun 24 '14 at 16:09
  • @Ramesh if you're going to reformat the drive (which you'll have to do to add more partitions), just reformat it with sufficient inodes. – derobert Jun 24 '14 at 16:10
  • I already reformatted the system. But as per the error, I believe I have to do it again. Anyways, I really appreciate your help. I can confidently tell people what seems to be the kill in time of this particular backup process. :) – Ramesh Jun 24 '14 at 16:12