12

I have a server which stores a lot of data, I keep many nested directories with millions of files in them.

While trying to move one big directory (~700GB) from one disk to another I get an error:

mv /media/storage3/dir /media/storage4/

"No space left on device"

Also, trying to create a small txt file on the disk does not work:

Error opening file '/media/storage4/Untitled Document': No space left on device

I checked many options online, none worked.

df -h output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdg7        77G  9.2G   64G  13% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev             32G  4.0K   32G   1% /dev
tmpfs           6.3G  1.4M  6.3G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none             32G   76K   32G   1% /run/shm
none            100M   48K  100M   1% /run/user
/dev/sdg6        19G  6.7G   11G  39% /home
/dev/sdb1       440G  278G  140G  67% /var
/dev/sda1       2.8T  2.0T  764G  73% /media/storage1
/dev/sdh1       2.8T  1.6T  1.3T  56% /media/storage2
/dev/sdi1       2.8T  1.7T  1.2T  60% /media/storage3
/dev/sdf1       2.8T  876G  1.9T  32% /media/storage4

df -i output:

Filesystem         Inodes    IUsed      IFree IUse% Mounted on
/dev/sdg7         5079040   514215    4564825   11% /
none              8242449        2    8242447    1% /sys/fs/cgroup
udev              8239750      611    8239139    1% /dev
tmpfs             8242449      696    8241753    1% /run
none              8242449        5    8242444    1% /run/lock
none              8242449        4    8242445    1% /run/shm
none              8242449       28    8242421    1% /run/user
/dev/sdg6         1220608   218613    1001995   18% /home
/dev/sdb1        29310976 12863877   16447099   44% /var
/dev/sda1       858436804 41630853  816805951    5% /media/storage1
/dev/sdh1      1356948436 38728057 1318220379    3% /media/storage2
/dev/sdi1      1217505624 34748869 1182756755    3% /media/storage3
/dev/sdf1      2048962648 36308921 2012653727    2% /media/storage4

Reading online, people said similar problems happen because of access to a huge amount of files and that setting fs.inotify.max_user_watches to a high amount might solve it, I increased it from 8192 to 1000000 but it did not help.

Jander
  • 16,682
PeterN
  • 221
  • Have you tried lsof +L1? man lsof: A specification of the form +L1 will select open files that have been unlinked. – A.B. Mar 26 '15 at 12:57
  • 1
    There is no reason for mv to use the inotify mechanism. Have a look here : some processes may still be using unlinked files from the destination filesystem. – John WH Smith Mar 26 '15 at 12:57
  • Deleted files would still show up in the usage given by df... – Stephen Kitt Mar 26 '15 at 13:00
  • @StephenKitt No. Unlinked files which are held by the file subsystem due to inode referencing will. Those which got completely unused and unlinked ("deleted") do not show up. See here. – John WH Smith Mar 26 '15 at 13:37
  • Yeah that's what I meant, I was referring to the files you were talking about: if they're unlinked but still in use, they still count against the usage shown by df. I.e. given that df show 1.9TB free on storage4, it really is possible to write 1.9TB there. – Stephen Kitt Mar 26 '15 at 13:55
  • I checked lsof +L1, there weren't many open files specified, most file were in temp dir, and there were no special files with exceptional SIZE/OFF. The largest one was: `COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME'

    'java 3745 peter 16r REG 8,103 3233439 0 1967773 /tmp/jar_cache1670618378288564032.tmp (deleted)` I also tried looking for open files as in the link @JohnWHSmith sent, didn't find anything suspicious. There was nothing open under storage4 or sdf1 (I used grep to filter these out)

     – PeterN Mar 27 '15 at 05:38
  • currently I am trying to delete the part of the dir which was moved to storage4 and to use rsync to copy the dir from storage3 to storage4 again, I will update you guys on how it goes – PeterN Mar 27 '15 at 05:45
  • Removing the dir and stating again did not work. The rsync comman started copying but hung after a couple of minutes. Trying to restart rsync did not help, after the third restart attempt I got a failed: No space left on device (28) message. Also, creating a simple text file on storage4 does not work: Error opening file '/media/storage4/Untitled Document': No space left on device – PeterN Mar 27 '15 at 07:11
  • If it is an option, have you tried rebooting between attempts? – JonathanS Mar 27 '15 at 23:54
  • Yes, several times. The drive is formatted to NTFS (I have dual boot with windows). I don't use the windows installation, but still I even tried windows disk repair and it did not work. I think I am going to backup the data and reformat the drive – PeterN Mar 30 '15 at 12:57
  • did the "disk full" drive somehow find its way to be mounted read only? sorry if this was too obvious and therefore excluded. – mitchell babies peters Apr 28 '15 at 22:15
  • Is it fuse-ntfs or ntfs-3g? – Costa May 11 '15 at 15:33
  • What is the underlying fs? If it is ext4, you may need to increase the inode limit with tune2fs. – wages May 07 '15 at 23:46
  • Would you please share with the /etc/mtab and your /proc/mount files so we can see the actual filesystem types as they are configured on your system. – Manchego May 21 '15 at 16:02
  • 1
    Check ntfsinfo -m /dev/DEV for clues. It's part of ntfs-3g. Also note that there can only be up to 2^32 files on a single NTFS volume. I wouldn't trust any native linux/unix tool to report NTFS inodes count precisely. – jficz Jun 03 '15 at 16:18
  • what does: lsof | grep deleted | grep media look like? – Stephan Jun 11 '15 at 22:43
  • All the tests are not relevant now because i have already formatted the disc. If it happens again I will check them and report to you guys – PeterN Jun 20 '15 at 14:05
  • How much free space does Windows say you have? – cybernard Aug 14 '15 at 19:55

3 Answers3

1

Run the following:

lsof -s | sort -nrk 7 | head

You'll see output like so:

firefox    2997                     j   52rr     REG              252,0 10485760    5505182 /some/path
firefox    2997                     j   50rr     REG              252,0 10485760    5505182 /some/path
firefox    2997                     j    3rr     REG              252,0 10485760    5505182 /some/path
firefox    2997                     j   39rr     REG              252,0 10485760    5505182 /some/path
firefox    2997                     j   31ur     REG              252,0 10485760    5505182 /some/path

Note; the above is just an example you'll need to use your judgement to find the space usage.

About output:

  • Column 7 is the size of the space in use.
  • Column 4 is the file descriptor.

You can remove the content at the file-descriptor to gain back that space:

cat /dev/null > /proc/2997/fd/50

Obviously this will destroy all data on that file descriptor.

0

You could try a tar to copy all the data, then remove it afterward:

cd /media/storage3/; tar cfp - ./dir | ( cd /media/storage4; tar xfp - )

This will preserve all ownership, permissions and links

Chad Smith
  • 176
  • No copy worked.. Not only that, creating a simple empty file returned no space left exception – PeterN Apr 23 '15 at 08:05
0

I would have used rsync -avrz storage3 storage4 and then deleted the files afterwards, but I got another question why do you have those shares using NTFS filesystem if you are using Linux (Ubuntu) then I would have gone for ext4 or ReiserFS, ButterFS. IF those shares are located on a Windows box, please try and move those files from Windows to Windows using robocopy, which is excellent in this regards. Robocopy works mostly like rsync.

man page for rsync: http://linux.die.net/man/1/rsync

help page for robocopy: https://technet.microsoft.com/en-us/library/cc733145.aspx

support page for 3g-ntfs drivers for Linux: www.tuxera.com/community/open-source-ntfs-3g/

FAQ for 3g-ntfs: www.tuxera.com/community/ntfs-3g-faq/

I suspect something should be fixed inside the NTFS filesystem drivers you are using in your Ubuntu installation, which distribution of NTFS are you using, how did you mount the disks, are they local attached to the ubuntu box you are sitting on or are they mounted using iSCSI from a SAN?

There is more to this troubleshooting than, just "I cannot move my files they are located on NTFS and I'm running ubuntu to move them."

OMG-1
  • 434
  • As mentioned, no copy worked, i tried all sorts of rsync commands. But, even creating an empty file on storage4 returned an exception. \n The reason for using NTFS is because we have other servers, which do run windows, and we wanted to allow for easy transfer of disks from one server to another. In retrospect, I wouldn't have used NTFS because of all the problems it brings with it, Ubuntu is very unstable with an NTFS mount. \n The disks are connected via SATA cable directly to the motherboard on the computer, no SAN or anything. It's a direct mount on the linux system. – PeterN Apr 23 '15 at 08:06
  • I would advice you to two things, when using MS technologies go pure MS, use a Microsoft Windows Fileserver if you need to have NTFS, else go with ext4 or other Linux filesystem and share the disk with Smb. – OMG-1 Apr 23 '15 at 14:43
  • Save yourself some problems and never try to combine windows and linux. No amount of help will save you in the long run, eventually this will always end in tears. It does seem like a problem with inode or file handles, but when you put NTFS in the mix... time bomb. If your environment needs interaction of linux and NTFS, put the NTFS on a windows server and do the interaction that way. Not trying to combine onto the same system. – Baazigar May 19 '15 at 18:42