40

And now I am unable to chmod it back.. or use any of my other system programs. Luckily this is on a VM I've been toying with, but is there any way to resolve this? The system is Ubuntu Server 12.10.

I have attempted to restart into recovery mode, unfortunately now I am unable to boot into the system at all due to permissions not granting some programs after init-bottom availability to run- the system just hangs. This is what I see:

Begin: Running /scripts/init-bottom ... done
[   37.062059] init: Failed to spawn friendly-recovery pre-start process: unable to execute: Permission denied
[   37.084744]  init: Failed to spawn friendly-recovery post-stop process: unable to execute: Permission denied
[   37.101333] init: plymouth main process (220) killed by ABRT signal

After this the computer hangs.

jett
  • 575

5 Answers5

73

Even as root, you can't execute files that have no x permission bit set. What you can do though is call ld.so on it (provided they're dynamically linked executables):

$ echo /lib/*/ld*.so
/lib/i386-linux-gnu/ld-2.27.so /lib/x86_64-linux-gnu/ld-2.27.so

Use the one that matches the architecture of chmod executable. In my case the x86_64 one:

sudo /lib/x86_64-linux-gnu/ld-2.27.so /bin/chmod 755 /bin /bin/chmod

Or call something in /usr/bin or elsewhere to do the chmod like perl:

sudo perl -e 'chmod 0755, "/bin", "/bin/chmod"

Beware when restoring permissions that some files in /bin like mount or su are meant to have permissions other than 0755.

If you've rebooted, however, you might not be able to get to the point where you can run perl or ld.so though. You can fix things from the initramfs though (pass an incorrect root directory to get a recovery shell in the initramfs; see also the break=bottom or break=init kernel parameter on Debian, for the initramfs to give you a shell after the root file system was mounted (read-only though)). Or boot your VM from a live CD image, or fix by mounting the VM file system on the host as others suggested.

Fixing the initramfs way:

In grub, edit the boot entry and remove the root= parameter from the linux command:

setparams 'Ubuntu, with Linux 3.2.0-27-generic'

recordfail
gfxmode $linux_gfx_mode
insmod gzio
insmod ext2
set root='(hd1)'
search --no-floppy --fs-uuid --set=root dc02b07c-88ef-4804-afe0-4f02db2\ 94561
linux /boot/vmlinuz-3.2.0-27-generic
initrd /boot/initrd.img-3.2.0-27-generic

Ctrl-X to boot. Ubuntu's initramfs won't find the root file system so start a recovery sh. Then mount the root filesystem (in my case /dev/vdb, adapt to your machine) and fix things there:

Target filesystem doesn't have requested /sbin/init.
No init found. Try passing init= bootarg.

BusyBox v1.18.5 (Ubuntu 1:1.18.5-1ubuntu4) built-in shell (ash) Enter 'help' for a list of built-in commands.

(initramfs) mkdir /x (initramfs) mount /dev/vdb /x [ 48.430071] EXT3-fs (vdb): error: couldn't mount because of unsupported optio nal features (240) [ 48.477406] EXT4-fs (vdb): recovery complete [ 48.477747] EXT4-fs (vdb): mounted filesystem with ordered data mode. Opts: ( null) (initramfs) chmod -R 755 /x/bin (initramfs) umount /x (initramfs) reboot

Once booted, fix the permissions of the files that are not meant to have 755 permissions by comparing with another system.

Fixing by running python as init:

In grub, edit the boot entry, this time keep the root= parameter, change ro to rw and add a init=/usr/bin/python:

setparams 'Ubuntu, with Linux 3.2.0-27-generic'

recordfail
gfxmode $linux_gfx_mode
insmod gzio
insmod ext2
set root='(hd1)'
search --no-floppy --fs-uuid --set=root dc02b07c-88ef-4804-afe0-4f02db2\ 94561
linux /boot/vmlinuz-3.2.0-27-generic root=UUID=dc02b07c-88ef-4804-afe0-
4f02db294561 rw init=/usr/bin/python initrd /boot/initrd.img-3.2.0-27-generic

Then, at the python prompt:

Begin: Running /scripts/init-bottom ... done.
Python 2.7.3 (default, Apr 20 2012, 22:39:59)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.chmod('/bin/sh',0755)
>>> os.chmod('/bin/chmod',0755)
>>> os.execl('/bin/sh','sh')
sh: 0: can't access tty; job control turned off
# chmod -R 0755 /bin
# mount -o remount,ro /
[  100.704720] EXT4-fs (vdb): re-mounted. Opts: errors=remount-ro
# exec /sbin/init

Again, once booted, fix the permissions of the files that are not meant to have 755 permissions by comparing with another system.

  • 5
    +1, another great answer, Stephane. I'd add: in the already try to reboot case : boot on a live CD, mount rw the partition containing /bin, and chmod 755 /bin (and files inside if they were changed too). But afterward, check that all files are the right permission (depending on your linux distrib, you probably can check /bin against the original package) – Olivier Dulac Jun 01 '13 at 07:29
  • 6
    WOW. Your depth of knowledge is scary 8-). – slm Jun 01 '13 at 16:52
  • I can't use ld.so to execute. http://pb.abhijeetr.com/fRWf What's going wrong here? – Abhijeet Rastogi Jun 04 '13 at 04:13
  • @shadyabhi, you probably have a multiarch system and are trying to use the 32bit ld.so on a 64bit executable. You must have another ld.so, maybe in some directory like /lib/x86_64-linux-gnu. – Stéphane Chazelas Jun 04 '13 at 06:32
  • @StephaneChazelas Thanks. It was indeed the case. The correct path in Archlinux is /usr/lib/ld-2.17.so – Abhijeet Rastogi Jun 04 '13 at 08:18
  • I suggest doing a reboot instead of running /sbin/init manually. Stuff might go apeshit otherwise (PID 1 = /usr/bin/python). – Chris Warrick Jun 08 '13 at 09:10
  • 1
    @Kwpolska, os.execl and exec are for executing, they don't fork a process, just replace the executable in the same process, so all is done in pid 1. Process 1 initially runs python, then sh, then init. – Stéphane Chazelas Jun 08 '13 at 09:29
30

Boot another clean OS, mount the file system and fix permissions.

As your broken file system lives in a VM, you should have your host system available and working. Mount your broken file system there and fix it.

In case of QEMU/KVM you can for example mount the file system using nbd.

michas
  • 21,510
  • I think this is probably correct as a way to fix it, however, I need to try to get the system mounted- right now I am just getting the presystem from the image file- as in initrd.img memtest & abi. – jett Jun 01 '13 at 07:25
  • 1
    @jett, you mounted the /boot partition of that VM. Try and locate the root file system. If on LVM, run vgchange -ay after connecting the nbd to activate it. – Stéphane Chazelas Jun 01 '13 at 07:44
  • 1
    @StephaneChazelas I've got it. Thank you both so much- I love these kinds of mistakes, have learnt a ton! – jett Jun 01 '13 at 08:08
  • Glad it's fixed. I'm missing something. If it's a vm, then it's just one big file to the host system, isn't it? How do you mount anything inside it to do the repairs? (I understand how to do this on a non-vm system.) – Joe Jun 07 '13 at 19:18
10

Use python :)

$ python
>>> import os
>>> os.chmod('/bin', 0755)

That shouldn't need anything from /bin to do its job. Obviously, I haven't tried this out...

  • Hmm, I always figured that this and other scripting languages would just call the chmod program. Pretty good to know! – jett Jun 01 '13 at 08:36
  • 3
    No, chmod is a system call, called by the chmod program and also by the chmod function in python/perl/ruby etc. shells do call the chmod utility though. – Dennis Kaarsemaker Jun 01 '13 at 08:43
  • 2
    Except those shells that have chmod builtin. That's the kind of situation where shells like sash are useful. It's statically linked and has most recovery commands like chmod builtin (so doesn't rely on anything else). It would usually sit in /sbin though it wouldn't harm having extra copies on all the file systems, and can be used in combination with memlockd. zsh and ksh93 have a chmod builtin (though not enabled by default). – Stéphane Chazelas Jun 01 '13 at 17:13
  • 1
    @Dennis But how can you run it if you cannot boot into the system at all? As the OP says: «unfortunately now I am unable to boot into the system at all». – Nadir Sampaoli Jun 02 '13 at 10:08
  • @NadirSampaoli This is definitely one of those things that you need to catch before you reboot. This could be successfully done after chmodding root and before shutting down the system. – Ken Bellows Jun 14 '13 at 18:39
1

First, copy the /bin/chmod file into /tmp:

install -m 775 /bin/chmod /tmp/chmod

then recover your permissions:

chmod -R 775 /bin
0

You can try sudo chmod -R 744 /path-to-your-system/bin from a live distro.