13

As the headline says everything or almost everything important as root under root (/) was moved to /old on a Solaris 10 machine. So now the typical fault when trying when running commands are Cannot find /usr/lib/ld.so.1 (changed $PATH and also tried changing $LD_LIBRARY_PATH, $LD_LIBRARY_PATH_64 and $LD_RUN_PATH and exporting them but nothing of that seems to change the real library path). Tried pretty much yesterday to find something that might help but found nothing that will actually change the library path for Solaris 10 other than maybe crle but can't run that since Cannot find /usr/lib/ld.so.1.

Found a lot of root or /usr/bin recovery tips and so on for Linux but the that information for that regarding Solaris 10/Unix is not rife and very sparse.

Can't run cp, ln, mkdir or mv since Cannot find /usr/lib/ld.so.1. Can't neither log in with other sessions to the machine. Though one session is still up which can be used and that window's being stalled with while true; do date; echo hej 1234567; done. We've discussed the solution to use a Solaris boot CD and also a Linux dist on a USB drive. We've discussed the solution to switch the hard drive disks to another rack.

The /.../static/.../mv solution has been tested but it didn't work.

The commands that still can be used are (there might be more commands that can be used): echo, <, >, >>, |, pwd, cd.

Is there a way to create a directory or folder without mkdir? Is there any way to use echo and > or echo and >> to restore /usr/lib/ld.so.1? I know that more than /usr/lib/ld.so.1 will probably need to be restored in order for commands to work.

Thank you very much for reading and have a very nice day =)

Gilles 'SO- stop being evil'
  • 829,060
propatience
  • 135

3 Answers3

19

If you no longer have a shell running as root, you'll have to reboot into rescue media. Anything will do as long as it's capable of mounting the root filesystem read-write.

If you can still run commands as root, everything's copacetic. Set the environment variable LD_LIBRARY_PATH to point to the directories containing libraries used by the basic system tools. That's at least /usr/lib on a 32-bit Solaris, /usr/lib/64 on a 64-bit Solaris, possibly other directories (I don't have access to Solaris 10 now to check). To run an executable, prefix it with the runtime linker: /usr/lib/ld.so.1 (for a 32-bit executable) or /usr/lib/64/ld.so.1 (for a 64-bit executable) — now moved to /old. Thus you should be able to recover with something like:

LD_LIBRARY_PATH=/old/usr/lib
export LD_LIBRARY_PATH
/old/usr/lib/ld.so.1 /old/usr/bin/mv /old/* /
Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
Gilles 'SO- stop being evil'
  • 829,060
  • Smart use of the loader. If the whole hierarchy wasn't moved to /old, my comment to Peschke own comment is valid here too. mv shouldn't be used but a command that preserve the existing hierarchy like pax, tar and the likes. – jlliagre Feb 21 '17 at 20:43
  • 1
    Thank you very much for your patience and taking time to write this. It works now running commands like /old/usr/lib/ld.so.1 /old/usr/bin/mv and /old/usr/lib/ld.so.1 /old/usr/bin/cp and so on. I'll see what I can do, now the problem is that I'm out of space but I might be able to fix that. – propatience Feb 22 '17 at 09:22
  • @propatience Too bad you and Gilles didn't considered my comments... – jlliagre Feb 22 '17 at 10:44
  • @jlliagre I still don't understand what you're on about with not using mv. The files were moved, so move them back. Why do you want to preserve the hierarchy under /old? – Gilles 'SO- stop being evil' Feb 22 '17 at 10:57
  • The issue is more than likely that not all the files were moved to /old. Otherwise, the OP wouldn't be struggling with cp. The mv command you suggested unsynchronized the top directories by creating bogus /xx/xx hierarchies. – jlliagre Feb 22 '17 at 11:02
  • @jlliagre Ah, ok. Well, yes, that would be a problem, but the question didn't give any indication that there were files left in /. I'd recommend rsync -a in that case if it's installed. – Gilles 'SO- stop being evil' Feb 22 '17 at 11:05
  • The OP wrote "everything or almost everything important". – jlliagre Feb 22 '17 at 11:06
  • and rsync might have filled the file system. Moreover -a isn't enough, should be -aHAX to make sure nothing is lost. – jlliagre Feb 22 '17 at 11:18
  • 1
    Yay GOD! I managed to move back everything with first /old/usr/lib/ld.so.1 /old/usr/bin/mv and /old/usr/lib/ld.so.1 /old/usr/bin/cp but no space so couldn't use /old/usr/lib/ld.so.1 /old/usr/bin/cp. And then editing $PATH and then folder by folder using mv with changed $PATH. Now it's all moved back so I can log in again and type df -h and everything. Thank you very much everyone. Really, great thanks to you @Gilles! Thank you StackExchange! – propatience Feb 22 '17 at 17:34
  • 1
    Setting LD_LIBRARY_PATH on Solaris has some perils if you need to set it so you can run both 32- and 64-bit executables. It'd be better to use the 32- and 64-bit specific versions of LD_LIBRARY_PATH that are available in Solaris. In this case: LD_LIBRARY_PATH_32=/old/usr/lib and LD_LIBRARY_PATH_64=/old/usr/lib/64. See the ld.so.1 man page: "Each environment variable can be specified with a _32 or _64 suffix. This makes the environment variable specific, respectively, to 32–bit or 64–bit processes." – Andrew Henle Mar 20 '18 at 15:14
16

There is no way to create a directory or copy binary files with just shell builtin commands (although Gilles describes a smart potential workaround in his reply).

You best option is to boot Solaris on a external media (dvd, usb stick), mount or import the file system(s), and fix the mess with something like:

  • Boot a Solaris installation disk and select to run a shell.

  • Mount the old root (and all other file systems if any) under some directory like /mnt or /a. The Solaris boot disk might help you doing that when it detects existing file systems.

  • put the files back to their original location with this command (assuming all is mounted under /mnt):

    cd /mnt/old
find . -depth | cpio -pdlmPV@ /mnt

  • Reboot your system 

    init 6

If you are using ZFS and a recent snapshot exists, backing up the /old directory elsewhere then reverting to the last snapshot might be also an option.

Community
  • 1
jlliagre
  • 61,204
  • Great answer. Boot into rescue USB, mount the filesystem, and then do something like mv /mountpoint/old/* /mountpoint – Peschke Feb 21 '17 at 14:51
  • 1
    @Peschke Thanks, although blindingly running your suggested mv command wouldn't be my advice. The OP wrote "everything or almost everything". In the second case, moving will break still existing directories. I likely use a utility like tar, pax, or cpio to preserve the destination directory hierarchy. – jlliagre Feb 21 '17 at 14:59
  • Tied to the possible snapshot (ZFS or UFS), you could boot to an alternate BE if you're using live upgrade. lustatus

    And don't panic yet. All the data still exists on the disk(s).

     – sleepyweasel Feb 21 '17 at 22:34
1

Solaris includes static builds of basic utilities (cp, ln, mv, rcp, and tar) in /usr/sbin/static that you can use to repair any problem with the availability of /usr/lib/ld.so.1 that prevents the use of the regular dynamically-linked /usr/bin versions.

There is no static mkdir provided, but you can use the static ln to symlink whatever directory contains lib/ld.so.1 into place as /usr temporarily, and then use the standard mkdir to create whatever directories you need. You can rename directories that already exist using the static mv.

This goes at least as far back as Solaris 2.5.1, where I found that the stock ld.so.1 segfaults if you try to run it manually as shown in Gilles' example.

Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
rakslice
  • 1,177