Unable to delete file, even as root

Question

I've got a file on remote machine (saying that, cause I don't know who the heck created it), that I've to remove.

user@machine:~/folder$ ls
lift_proto.db.lock.db

And permissions are like this:

user@machine:~/folder$ ls -al
total 12
drwxrwxrwx  2 root          root         4096 2012-03-06 20:57 .
drwxr-xr-x 26 user          group        4096 2012-03-06 20:53 ..
-rw-r--r--  1 root          root          126 2012-03-06 20:57 lift_proto.db.lock.db

Trying to delete:

user@machine:~/folder$ sudo rm lift_proto.db.lock.db

And nothing happens:

user@machine:~/folder$ ls
lift_proto.db.lock.db

I've seen that question, but this doesn't helped a lot:

user@machine:~/folder$ lsattr
-----------------e- ./lift_proto.db.lock.db

Moreover, I've successively tried to change ownership of file to user and permissions to 777, but then I'm deleting that file and it is still here back with root ownership and old permissions.

My last guess that somebody somehow syncing that file from elsewhere (btw, is there any way to see that?), if not (e.g. you're seeing that there is something with attrs or etc), how can I delete that?

UPDATE 1:

-f flag of rm doesn't solve problem
UPDATE 2: Here is the output of stat before and after:

user@machine:~/folder$ stat lift_proto.db.lock.db 
  File: `lift_proto.db.lock.db'
  Size: 126         Blocks: 8          IO Block: 4096   regular file
Device: 900h/2304d  Inode: 20054245    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-03-06 21:30:40.600796566 +0400
Modify: 2012-03-06 21:30:40.600796566 +0400
Change: 2012-03-06 21:30:40.600796566 +0400
user@machine:~/folder$ sudo rm -f lift_proto.db.lock.db 
user@machine:~/folder$ stat lift_proto.db.lock.db 
  File: `lift_proto.db.lock.db'
  Size: 126         Blocks: 8          IO Block: 4096   regular file
Device: 900h/2304d  Inode: 20054245    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-03-06 21:32:36.621421452 +0400
Modify: 2012-03-06 21:32:36.621421452 +0400
Change: 2012-03-06 21:32:36.621421452 +0400

UPDATE 3

strace output

user@machine:~/folder$ sudo strace rm -f lift_proto.db.lock.db

    execve("/bin/rm", ["rm", "-f", "lift_proto.db.lock.db"], [/* 14 vars */]) = 0
brk(0)                                  = 0x8c9000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc967443000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=42930, ...}) = 0
mmap(NULL, 42930, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc967438000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\355\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1572232, ...}) = 0
mmap(NULL, 3680296, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc966ea3000
mprotect(0x7fc96701d000, 2093056, PROT_NONE) = 0
mmap(0x7fc96721c000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x179000) = 0x7fc96721c000
mmap(0x7fc967221000, 18472, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fc967221000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc967437000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc967436000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc967435000
arch_prctl(ARCH_SET_FS, 0x7fc967436700) = 0
mprotect(0x7fc96721c000, 16384, PROT_READ) = 0
mprotect(0x60e000, 4096, PROT_READ)     = 0
mprotect(0x7fc967445000, 4096, PROT_READ) = 0
munmap(0x7fc967438000, 42930)           = 0
brk(0)                                  = 0x8c9000
brk(0x8ea000)                           = 0x8ea000
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2027984, ...}) = 0
mmap(NULL, 2027984, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc966cb3000
close(3)                                = 0
open("/usr/share/locale/locale.alias", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2570, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc967442000
read(3, "# Locale name alias data base.\n#"..., 4096) = 2570
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7fc967442000, 4096)            = 0
open("/usr/lib/locale/en_US.UTF-8/LC_IDENTIFICATION", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_IDENTIFICATION", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=373, ...}) = 0
mmap(NULL, 373, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc967442000
close(3)                                = 0
open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=26048, ...}) = 0
mmap(NULL, 26048, PROT_READ, MAP_SHARED, 3, 0) = 0x7fc96743b000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_MEASUREMENT", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_MEASUREMENT", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=23, ...}) = 0
mmap(NULL, 23, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc96743a000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_TELEPHONE", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_TELEPHONE", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=59, ...}) = 0
mmap(NULL, 59, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc967439000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_ADDRESS", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_ADDRESS", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=155, ...}) = 0
mmap(NULL, 155, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc967438000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_NAME", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_NAME", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=77, ...}) = 0
mmap(NULL, 77, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc967434000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_PAPER", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_PAPER", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=34, ...}) = 0
mmap(NULL, 34, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc967433000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_MESSAGES", O_RDONLY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
close(3)                                = 0
open("/usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=57, ...}) = 0
mmap(NULL, 57, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc967432000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_MONETARY", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_MONETARY", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=286, ...}) = 0
mmap(NULL, 286, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc967431000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_COLLATE", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_COLLATE", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1170770, ...}) = 0
mmap(NULL, 1170770, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc967313000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_TIME", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_TIME", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2454, ...}) = 0
mmap(NULL, 2454, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc967312000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_NUMERIC", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_NUMERIC", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=54, ...}) = 0
mmap(NULL, 54, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc967311000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_CTYPE", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_CTYPE", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=256324, ...}) = 0
mmap(NULL, 256324, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc9672d2000
close(3)                                = 0
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
unlinkat(AT_FDCWD, "lift_proto.db.lock.db", 0) = 0
close(0)                                = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?

Does the inode change after you have deleted it? What is the output of stat lift_proto.db.lock.db before and after you delete it? What about lsof lift_proto.db.lock.db? — Chris Down, Mar 06 '12 at 17:31
Is this on NFS? With what mount options? Can you access the server? — Gilles 'SO- stop being evil', Mar 06 '12 at 17:39
@Gilles /dev/md0 on / type ext4 (rw,errors=remount-ro), ssh access. — om-nom-nom, Mar 06 '12 at 17:43
If it were synced via rsync, the inode would change. The output of strace should prove to be interesting... — Kyle Smith, Mar 06 '12 at 17:43
@om-nom-nom sudo strace rm -f lift_proto.db.lock.db. You can't do strace sudo … as sudo can't gain its privileges when traced. — Gilles 'SO- stop being evil', Mar 06 '12 at 17:58

Mat · Accepted Answer · 2012-03-06T18:10:37.743

The file is an H2 file lock, from a lift-based (web) application. H2 uses an interesting file locking protocol, the file will be re-created pretty much immediately if that database is in use.

(The filename matches the default persistence database name for that framework.)

You need to stop whatever app server is running that database to get rid of that file if you want it gone. (But do you really?)

Your strace output clearly shows that the unlink is successful.

unlinkat(AT_FDCWD, "lift_proto.db.lock.db", 0) = 0

An inode number can be re-used. Filesystems could use whatever allocation strategy they want, but nothing prevents them from reallocating the same inode number.

On an idle ext3 filesystem:

$ touch a
$ stat -c %i a
593358
$ rm a
$ touch a
$ stat -c %i a
593358
$ touch a b
$ stat -c %i a b
593358
593362
$ rm a b
$ touch b a
$ stat -c %i a b
593362
593358

Oh, nice. But if this file was recreated, shouldn't inode been changed? — om-nom-nom, Mar 06 '12 at 17:51
The inode number can be re-used. Try it out yourself on an idle filesystem. — Mat, Mar 06 '12 at 17:54

score 0 · Answer 2 · edited Sep 22 '12 at 14:28

0

Did you try to delete this with inode value? This is explained in the following tutorial: How to delete a file in linux with spaces

Essentially, you retrieve a file's inode with ls -il (the inode is the first number in the listing, then you can use:

find . -inum 123456 -exec rm -i {} \;

(Replace 123456 by the inode number you found with ls -il.)

edited Sep 22 '12 at 14:28

Mat

52,586

answered Sep 08 '12 at 14:50

arbab

11

3

Please don't post a link, at least brief about what it talks about. – daisy Sep 08 '12 at 15:38

Unable to delete file, even as root

2 Answers2

Linked