Unable to add or delete users

Question

I'm currently having issues running adding, deleting and modifying Linux users. Reasonably new to Linux so pretty sure I have tried everything to resolve this.

[root@device01 /]# useradd testuser 
useradd: cannot lock /etc/passwd; try again later.

Attempting to change an existing users password also returns the follows:

[root@device01 ~]# passwd olduser 
Changing password for user olduser. 
New password:
BAD PASSWORD: it is based on a dictionary word 
Retype new password: passwd: Authentication token manipulation error

Have checked to make sure there are no lock files (/etc/gshadow.lock, /etc/shadow.lock, /etc/passwd.lock and /etc/group.lock)

I have tried running the following to make sure that the filesystem was mounted correctly which also failed:

[root@device01 /]# mount -o remount,rw / 
can't create lock file /etc/mtab~60598: No space left on device (use -n flag to override) 
[root@device01 /]#

Checking the diskspace usage doesn't reveal anything (unless I'm missing something)

[root@device01 /]# df -k 

Filesystem            1K-blocks       Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root 51606140    6953044  42031656  15% / 
tmpfs                   3881396          0   3881396   0% /dev/shm
/dev/mapper/ddf1_4c5349202020202080862925000000004711471100001e78p1
                             495844     120559    349685  26% /boot
/dev/mapper/VolGroup-lv_home
                          112173808   20244156  86231504  20% /home
/dev/mapper/ddf1_4c53492020202020808629250000000047114711000028a0p1
                         1032131696  382893688 596808680  40% /backup
/dev/sde1            1922828276 1011928116 813226168  56% /mnt/BackupDrive

The server has not been rebooted for around two weeks, and ignoring the golden rule would prefer to use this as a last resort.

Version information is as follows:

[root@device01 /]# uname Linux [root@device01 /]# lsb_release LSB Version:
:base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch

Is there anything else I can try or am possibly missing?

Edit #1

[root@device01 /]# mount
/dev/mapper/VolGroup-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/ddf1_4c5349202020202080862925000000004711471100001e78p1 on /boot type ext4 (rw)
/dev/mapper/VolGroup-lv_home on /home type ext4 (rw)
/dev/mapper/ddf1_4c53492020202020808629250000000047114711000028a0p1 on /backup type ext4 (rw)
/dev/sde1 on /mnt/BackupDrive type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

Edit #2

[root@device01 ~]# strace useradd testuser
execve("/usr/sbin/useradd", ["useradd", "testuser"], [/* 28 vars */]) = 0
brk(0)                                  = 0x7fabaf304000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae831000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=48359, ...}) = 0
mmap(NULL, 48359, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fabae825000
close(3)                                = 0
open("/lib64/libaudit.so.1", O_RDONLY)  = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260%@\3243\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=115536, ...}) = 0
mmap(NULL, 2208304, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fabae3f7000
mprotect(0x7fabae40e000, 2093056, PROT_NONE) = 0
mmap(0x7fabae60d000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7fabae60d000
close(3)                                = 0
open("/lib64/libselinux.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0PX\300\3163\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=124624, ...}) = 0
mmap(NULL, 2221912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fabae1d8000
mprotect(0x7fabae1f5000, 2093056, PROT_NONE) = 0
mmap(0x7fabae3f4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x7fabae3f4000
mmap(0x7fabae3f6000, 1880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fabae3f6000
close(3)                                = 0
open("/lib64/libacl.so.1", O_RDONLY)    = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\36\200\3213\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=33816, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae824000
mmap(NULL, 2126416, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fabadfd0000
mprotect(0x7fabadfd7000, 2093056, PROT_NONE) = 0
mmap(0x7fabae1d6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7fabae1d6000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY)      = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\356A\3153\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1926520, ...}) = 0
mmap(NULL, 3750152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fabadc3c000
mprotect(0x7fabaddc6000, 2097152, PROT_NONE) = 0
mmap(0x7fabadfc6000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18a000) = 0x7fabadfc6000
mmap(0x7fabadfcb000, 18696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fabadfcb000
close(3)                                = 0
open("/lib64/libdl.so.2", O_RDONLY)     = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\r\300\3153\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=22536, ...}) = 0
mmap(NULL, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fabada38000
mprotect(0x7fabada3a000, 2097152, PROT_NONE) = 0
mmap(0x7fabadc3a000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fabadc3a000
close(3)                                = 0
open("/lib64/libattr.so.1", O_RDONLY)   = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\23\300\3203\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=21152, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae823000
mmap(NULL, 2113888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fabad833000
mprotect(0x7fabad837000, 2093056, PROT_NONE) = 0
mmap(0x7fabada36000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7fabada36000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae822000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae820000
arch_prctl(ARCH_SET_FS, 0x7fabae8207a0) = 0
mprotect(0x7fabada36000, 4096, PROT_READ) = 0
mprotect(0x7fabadc3a000, 4096, PROT_READ) = 0
mprotect(0x7fabadfc6000, 16384, PROT_READ) = 0
mprotect(0x7fabae1d6000, 4096, PROT_READ) = 0
mprotect(0x7fabae3f4000, 4096, PROT_READ) = 0
mprotect(0x7fabae60d000, 4096, PROT_READ) = 0
mprotect(0x7fabaea4b000, 4096, PROT_READ) = 0
mprotect(0x7fabae832000, 4096, PROT_READ) = 0
munmap(0x7fabae825000, 48359)           = 0
statfs("/selinux", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=12901535, f_bfree=11155729, f_bavail=10500369, f_files=3276800, f_ffree=0, f_fsid={1688549107, 1017566797}, f_namelen=255, f_frsize=4096}) = 0
brk(0)                                  = 0x7fabaf304000
brk(0x7fabaf325000)                     = 0x7fabaf325000
open("/proc/filesystems", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae830000
read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tb"..., 1024) = 317
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7fabae830000, 4096)            = 0
socket(PF_NETLINK, SOCK_RAW, 9)         = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
open("/usr/lib/locale/locale-archive", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=99158576, ...}) = 0
mmap(NULL, 99158576, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7faba79a2000
close(4)                                = 0
open("/proc/sys/kernel/ngroups_max", O_RDONLY) = 4
read(4, "65536\n", 31)                  = 6
close(4)                                = 0
mmap(NULL, 528384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae79f000
access("/etc/shadow", F_OK)             = 0
access("/etc/gshadow", F_OK)            = 0
open("/etc/default/useradd", O_RDONLY)  = 4
fstat(4, {st_mode=S_IFREG|0600, st_size=119, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae830000
read(4, "# useradd defaults file\nGROUP=10"..., 4096) = 119
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(5)                                = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(5)                                = 0
open("/etc/nsswitch.conf", O_RDONLY)    = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=1688, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae82f000
read(5, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1688
read(5, "", 4096)                       = 0
close(5)                                = 0
munmap(0x7fabae82f000, 4096)            = 0
open("/etc/ld.so.cache", O_RDONLY)      = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=48359, ...}) = 0
mmap(NULL, 48359, PROT_READ, MAP_PRIVATE, 5, 0) = 0x7fabae793000
close(5)                                = 0
open("/lib64/libnss_files.so.2", O_RDONLY) = 5
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360!\0\0\0\0\0\0"..., 832) = 832
fstat(5, {st_mode=S_IFREG|0755, st_size=65928, ...}) = 0
mmap(NULL, 2151824, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x7faba7794000
mprotect(0x7faba77a0000, 2097152, PROT_NONE) = 0
mmap(0x7faba79a0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0xc000) = 0x7faba79a0000
close(5)                                = 0
mprotect(0x7faba79a0000, 4096, PROT_READ) = 0
munmap(0x7fabae793000, 48359)           = 0
open("/etc/group", O_RDONLY|O_CLOEXEC)  = 5
fcntl(5, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
fstat(5, {st_mode=S_IFREG|0644, st_size=2595, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae82f000
read(5, "root:x:0:\nbin:x:1:bin,daemon\ndae"..., 4096) = 2595
close(5)                                = 0
munmap(0x7fabae82f000, 4096)            = 0
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0x7fabae830000, 4096)            = 0
open("/etc/login.defs", O_RDONLY)       = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=1816, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae830000
read(4, "#\n# Please note that the paramet"..., 4096) = 1816
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0x7fabae830000, 4096)            = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
connect(4, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(4)                                = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
connect(4, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(4)                                = 0
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=4671, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae830000
read(4, "root:x:0:0:root:/root:/bin/bash\n"..., 4096) = 4096
read(4, "b:x:560:501::/home/claireb:/bin/"..., 4096) = 575
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0x7fabae830000, 4096)            = 0
open("/etc/group", O_RDONLY|O_CLOEXEC)  = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=2595, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae830000
read(4, "root:x:0:\nbin:x:1:bin,daemon\ndae"..., 4096) = 2595
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0x7fabae830000, 4096)            = 0
open("/etc/.pwd.lock", O_WRONLY|O_CREAT|O_CLOEXEC, 0600) = 4
fcntl(4, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
rt_sigaction(SIGALRM, {0x7fabadd2b110, ~[], SA_RESTORER, 0x7fabadc6e6a0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [ALRM], [], 8) = 0
alarm(15)                               = 0
fcntl(4, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0
alarm(0)                                = 15
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigaction(SIGALRM, {SIG_DFL, [], SA_RESTORER, 0x7fabadc6e6a0}, NULL, 8) = 0
getpid()                                = 63618
open("/etc/passwd.63618", O_WRONLY|O_CREAT|O_EXCL, 0600) = -1 ENOSPC (No space left on device)
close(4)                                = 0
open("/usr/share/locale/locale.alias", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fabae830000
read(4, "# Locale name alias data base.\n#"..., 4096) = 2512
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0x7fabae830000, 4096)            = 0
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "useradd: cannot lock /etc/passwd"..., 51useradd: cannot lock /etc/passwd; try again later.
) = 51
exit_group(1)                           = ?

Edit #3

[root@device01 ~]# df -i /
Filesystem            Inodes   IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
                     3276800 3276798     2  100% /

What are the permissions on /etc? (ls -ld /etc). Wondering if it's got an odd permission then even root may not like it… — forquare, Jul 20 '16 at 06:35
The error from passwd happens before it tries to modify /etc/passwd so it's irrelevant. Run strace useradd testuser and see what it does. Post the full output (except for actual user names you want to keep private if it goes as far as reading them) if you need help to understand the trace. — Gilles 'SO- stop being evil', Jul 20 '16 at 20:47
@Gilles Same issue again, "useradd: cannot lock /etc/passwd" I must be missing a lock file somewhere but cant find it anywhere. (Added in the results from strace) — Michael Grubey, Jul 20 '16 at 23:21
@MichaelGrubey: To validate Gilles's hypothesis, can you show us df -i? — Julie Pelletier, Jul 21 '16 at 03:48
@JuliePelletier Gilles was correct, now to find how to get this reduced...thank you :). — Michael Grubey, Jul 21 '16 at 06:06
The simplest solution is to create a loop partition to hold a vast amount of preferably small files. — Julie Pelletier, Jul 21 '16 at 06:39

score 1 · Accepted Answer · edited Jun 11 '20 at 14:16

can't create lock file /etc/mtab~60598: No space left on device (use -n flag to override)
open("/etc/passwd.63618", O_WRONLY|O_CREAT|O_EXCL, 0600) = -1 ENOSPC (No space left on device)

It's definitely a disk space problem. Most filesystems have two limits: a limit on file contents, and a limit on the number of files — the size of the inode table. Since df / shows plenty of space left, by a process of elimination, the problem must be with the number of inodes. You can check with df -i /.

With ext4 (the most common filesystem type on Linux), there is no way to increase the number of inodes on an existing filesystem. If you have a lot of small files somewhere (How to report number of files in all subdirectories? may help), try to move them to another filesystem. You can shrink an ext4 filesystem to make room for another filesystem with more inodes (you can set the number of inodes when you create the filesystem) but this can't be done while the filesystem is mounted.

df -i / shown in question. You were correct...partition is full thank you. — Michael Grubey, Jul 21 '16 at 06:07

Unable to add or delete users

1 Answers1