What are your favorite painful Unix moments

Question

How about sharing your favorite lessons learned moments?

Funny how many of the answers involve the rm command. Or sad. — Barry Brown, Aug 31 '11 at 17:09

score 67 · Answer 1 · edited Jan 14 '12 at 03:56

67

I was curious if chmod 000 / would work.

Well, flawlessly. A few minutes later I was searching for a rescue CD.

edited Jan 14 '12 at 03:56

Kevin

40,767

answered Oct 10 '10 at 12:46

sheerun

335

11

Wow I never even considered that until you mentioned it. Good one. – Stefan Lasiewski Oct 11 '10 at 17:57
What happens if you do it on a regular file ? – Luc M Nov 16 '10 at 15:08
@Luc I guess you just need to chmod it back – phunehehe Nov 20 '10 at 16:07
I usually do such 'experiments' in my Arch VM. If it bombs I just roll it back. – André Paramés Jan 20 '11 at 23:47
I made this mistake once. It was highly embarrassing, but a valuable lesson. – Charles Sprayberry Jul 12 '11 at 03:38

score 37 · Answer 2 · edited Jan 14 '12 at 03:58

37

When I first started working as a user consultant for the university I was attending, I was given limited sudo rights to help students who had lost/forgotten their passwords. sudo passwd <username> was my new friend. An hour after my orientation, my curiosity got the better of me and I typed in sudo passwd and stared in horror at the prompt for a new password. I was a bit scared to ^C my way out of it, thinking (erroniously, it turns out) that I might leave the account in question in a transient state, so I entered a password and immediately walked upstairs to the hallowed 2nd floor domain of the campus SuperUser and asked if he would like to know the root password of the main system.

edited Jan 14 '12 at 03:58

Kevin

40,767

answered Aug 14 '10 at 21:16

Larry Smithmier

465

4

heh, ownage! Confused Deputy :P – Spudd86 Aug 17 '10 at 19:33
30

you could have entered a mismatch password at the 2nd confirmation prompt, so nothing would have been affected then and passwd would exit. – Wadih M. Sep 01 '10 at 14:53
1

@Wadih: passwd behaves funny when run as root. For example, when failing the typo check, it asks again. – György Andrasek Mar 06 '11 at 03:57
You could have just changed it back to what it was before. – HandyGandy Mar 10 '11 at 05:24
1

@HandyGandy he didn't know the original root password. He just had sudo rights to run passwd. – gnud Mar 10 '11 at 14:28

score 32 · Answer 3 · edited Jan 14 '12 at 03:58

32

Surprised nobody else has mentioned this one yet:

rm -rf .*

(While attempting to remove all hidden files and subdirectories, completely forgetting that it will recurse into . and ..)

edited Jan 14 '12 at 03:58

Kevin

40,767

answered Aug 20 '10 at 00:58

Mark

101

3

While that has burned me in the past, many versions of rm won't do that now. I tried on Darwin and got the error rm: "." and ".." may not be removed. – Stefan Lasiewski Aug 24 '10 at 03:43
7

I do that all the time, on all kinds of platforms. Works as intended on OpenBSD, NetBSD, Linux, and OpenSolaris. – polemon Aug 29 '10 at 13:47
If you want to match all dotfiles except for . and .., use .[^.]*. (Well, this will actually miss all files starting with .., but usually there is only one.) – Sven Marnach Jun 04 '11 at 23:38
@Sven: Another way is .??*, which I find easier to type. This one won't match two-letter dot files like .a, but those are unusual too. I search the config files in my home directory with grep -r .??*, for example. – Neil Mayhew Jul 05 '11 at 03:38

score 31 · Answer 4 · answered Aug 15 '10 at 23:27

31

Makefile:

clean:
    @rm -f * .o

Which, of course, makes make clean wipe your source code instead of just object files.

Lesson: use version control.

answered Aug 15 '10 at 23:27

Alex B

4,478

11

If you can't see it, notice the space character between * and .o – Denilson Sá Maia Aug 17 '10 at 23:32
1

I feel your pain, happened to all of us at some point :) – axel_c Aug 20 '10 at 07:32
Lesson reloaded: whenever you do shell-stuff, and you see a "", be extra, no, be extra* Extra EXTRA careful. – Jürgen A. Erhard Apr 12 '11 at 23:05
I did that one, by hand, after pulling an all-nighter in the lab. The project was due a few hours later. It turned out to be a blessing in disguise; the new, rewritten version I turned in was much better than the bandaged version I erased. – Barry Brown Aug 31 '11 at 17:07
1

Obligatory Bumblebee reference: https://github.com/MrMEEE/bumblebee/commit/a047be – Residuum Sep 05 '11 at 13:05

score 30 · Answer 5 · edited Sep 15 '11 at 12:46

30

Had a friend run :() { :|:&}; : on a remote server where we didn't have console access to. Couldn't reboot it, completely frozen, production server.

Broken down (by request) to make it a bit more readable.

:() # Define ':' as a function. Every time we say ':' execute the following code block
{ # Start of code block
    : # Call ':' again. 
    | # Pipe output to...
    : # Another ':' 
    & # Disown process. 
    # All on one line this would read :|:&, 
} # End of code block
; # End definition of ':' as a function
: # Call ':'

It might be easier to look at it as

bomb() { bomb|bomb& }; bomb

edited Sep 15 '11 at 12:46

manatwork

31,277

answered Aug 16 '10 at 14:13

Josh K

3,946

1

Would You be so kind to explain what it does? I can't figure it out and I don't want to try it ;) – naugtur Aug 19 '10 at 08:54
23

Look's like a fork bomb. Don't try this unless you really know what you're doing. – Zaid Aug 19 '10 at 09:29
5

Yup. It's a forkbomb. And a nice one :) It seems to fork bash interpreter as it endlessly tries to parse it. If You skip the last : it does nothing. And it doesn't use memory at all, just forks a lot. [Yes, I did try it:)]. Effects can be blocked with a quota on number of processes per user. – naugtur Aug 19 '10 at 10:16
8

http://www.cyberciti.biz/faq/understanding-bash-fork-bomb/ – naugtur Aug 19 '10 at 10:51
4

+1 for the above comment. It's far easier to understand than it looks. – Umang Aug 19 '10 at 11:54
1

had a friend run the formk-bomb while we were in the labs, doing assignments, completely messed all of my days unsaved work sob – phoenix24 Aug 19 '10 at 23:04

score 29 · Answer 6 · edited Jan 14 '12 at 03:59

29

I meant well, I really did. Trying to chmod recursively a directory and ended up swapping ./ with /.

As root of course, because only with root can true pain (and thus enlightenment) be achieved.

edited Jan 14 '12 at 03:59

Kevin

40,767

answered Aug 20 '10 at 01:13

Sygo

336

23

+1 for quotable "Only with root can true pain (and thus enlightenment) be achieved." – Stefan Lasiewski Aug 27 '10 at 22:49

score 21 · Answer 7 · edited Jan 14 '12 at 04:00

21

I wiped the partition table of my main drive by accident, thinking I was working on another drive.

With scrollback, careful use of df, memory, and luck I was able to recreate it exactly, rewrite it, reboot, and hope... And it worked.

edited Jan 14 '12 at 04:00

Kevin

40,767

answered Sep 01 '10 at 19:28

Tom Ritter

141

6

A friend did this, and I helped him restore the table. Unfortunately, we didn't have the previous values for the partition sizes and offsets, so we used a bash loop with dd reading the first 4k block of every cylinder piped into file - to find the superblock and thus the start of the filesystem. This was on a live CD and there wasn't enough RAM to do everything we needed to do (which included installing a package or two) so we piped into a process running in ssh on another machine. – Neil Mayhew Sep 12 '10 at 23:37
2

Ouch. That's why I always use sfdisk -O to back up the partition table, always. FYI: http://www.cgsecurity.org/wiki/TestDisk can automate what @Neil Mayhew's did. – ephemient Nov 17 '10 at 18:25
2

I was there too, testdisk saved my box – phunehehe Nov 20 '10 at 16:12
These days, we have gpart, which looks for things that might resemble filesystems and constructs a partition table from that. – Simon Richter Mar 10 '11 at 17:54

score 19 · Answer 8 · edited Jan 03 '11 at 18:04

19

Not really my moment, but someone else's.

Back when I worked at a nuclear sciences research facility we used to run a number of SunOS, Ultrix and Linux computers and researchers had to share the CPU on those machines. As individual research groups got their own research grants they purchased their own computers, mostly SparcStations and they did the system administration themselves.

SunOS used to ship with the OpenView desktop and a nice file manager, this is what it looked like: alt text

Most of our researchers were running as root, and more than once we had to reinstall their operating systems because someone had decided to tidy-up the root directory and moved /bin, /etc, /tmp and everything else that cluttered the view into either the Trashcan or some subfolder.

Other users chose to tidy up the /bin directory and remove any command they did not know.

The lucky ones had back ups, most had purchased a tape drive, but did not have a tradition of running backups themselves.

edited Jan 03 '11 at 18:04

Josh

8,449

answered Aug 15 '10 at 16:07

miguel.de.icaza

5,319

11

Tidy up the root directory? Really? REALLY? *sob* – Alex B Aug 16 '10 at 03:54
41

TRWTF is they were running as root, at a nuclear sciences research facility! – invert Aug 16 '10 at 08:37
4

cool screenshot! makes me all nostalgic. – gabe. Aug 16 '10 at 16:17
@Alex - well there was this big VMUNIX file that I wasn't using .... – Martin Beckett Aug 20 '10 at 00:10
1

On old Mac OS versions, you could pretty much move any file anywhere, so maybe that is what they were expecting... – Kevin Cantu Oct 14 '10 at 21:01
@Kevin: Funny, that's one of my first experience with computers. At age 8 or so, I decided to "clean up" the System disk of my Apple IIGS. Sure, I could move any file anywhere... once. AFter I rebooted I couldn't use the computer anymore until I found a friend with an Apple IIGS and cloned his Systen disk :-) – Josh Jan 27 '11 at 13:34

score 18 · Answer 9 · edited Jan 14 '12 at 04:05

18

rm -f * ~

and

rm -rf ${DIR}/

when DIR was not set!

edited Jan 14 '12 at 04:05

Kevin

40,767

answered Aug 24 '10 at 01:52

Ivan P

119

4

You mean ${DIR}? Because $(DIR) would try to execute DIR command. – Denilson Sá Maia Nov 13 '10 at 22:49

score 18 · Answer 10 · answered Aug 14 '10 at 21:28

18

Back in the mid to late 90s, a friend of mine and I were discussing the folly of rm -rf * and at what point a Linux box would go belly up. We got into statically linked versus dynamically linked libraries and I posited that the system could live quite well without /lib and then proceeded to rename it on my workstation. Bad things happened, but we were left with several open console windows with which to try and fix the damage (shutdown wasn't an option anymore). None of the editors would run. It is amazing the esoteric uses you can find for the echo command.

answered Aug 14 '10 at 21:28

Larry Smithmier

465

19

There is a classic story on this exact topic: http://lug.wsu.edu/node/414 – Alex B Aug 16 '10 at 10:19
1

@Alex B: The story is also part of the Unix Horror Stories, which is just great to read. – Bobby Feb 09 '11 at 19:50

score 18 · Answer 11 · edited Jan 14 '12 at 04:04

vi and Caps-Lock vs. /etc/passwd

Connect to an old Solaris box using an old serial terminal that doesn't refresh the screen correctly.
su -
vi /etc/passwd. There is no vipw, and "we're just making minor edits" anyways.
Hit Caps-Lock key and don't notice.
Hit j a couple times to scroll down. Ignore the fact that you actually just typed J ("Join"), which combines this line with the next line. The serial terminal screen was not refreshing correctly, so you didn't see that you just combined the first 5 lines into one Loooooong line, thereby corrupting the first 5 users ('root', 'daemon', etc).
Finish your OTHER intended edits to the file, way down at the bottom.
Save file.
Log out.

I did this once. Amazingly, the system remained functional for months. Cronjobs ran fine, no errors stood out in the logfiles.

We didn't notice this problem until we rebooted the system months later and couldn't log in at the console. ps showed a bunch of jobs owned by UID '0' not by user 'root'.

You could not log in as root, nor run su or su -, and there was no sudo on this box. There was no floppy drive, the CD-ROM was busted and no USB ports (so no external CD-ROM). Single user mode did not work, because you need to type in the password for root, and that comes from /etc/passwd.

Surely "J" is the "Join" (as in join lines together) command? — dr-jan, Aug 19 '10 at 18:07
Wait a minute, I don't need to type the password to boot into single user mode. In fact that is how I reset the root password if I lose it. Or was it different in the old time? — phunehehe, Nov 20 '10 at 16:05
On Solaris, you need to know the root password to enter single user mode. I had to do this 2 weeks ago actually. — Stefan Lasiewski, Nov 22 '10 at 05:38
@phunehehe I think your thinking of the init=/bin/sh trick. That does indeed not require the root password. However runlevel 1 does still require root pass — phemmer, Feb 07 '11 at 16:16

score 16 · Answer 12 · edited Jan 14 '12 at 04:06

16

I remember trying to send a SysRq key sequence to a remote machine...

...but it was captured by the local one.

edited Jan 14 '12 at 04:06

Kevin

40,767

answered Sep 02 '10 at 09:33

polemon

11,431

score 14 · Answer 13 · answered Aug 31 '10 at 10:16

14

A simple halt recognizing some seconds later that I'm not on a local shell and having no possibility to power on the production server again.

Lessons learned? The prompt of the machine now looks like

[ --> root <-- @kompost:/home/echox] #

with some nice red markup ;-)

answered Aug 31 '10 at 10:16

echox

18,103

6

There is a tool called molly-guard that checks whether you are logged in remotely and asks whether you really want to do this. – Simon Richter Mar 10 '11 at 17:58
IPMI is a really convenient thing to have :-) – Andre Holzner Aug 30 '11 at 19:40

score 13 · Answer 14 · answered Jan 21 '11 at 21:09

13

My favourite moment was, when a co-worker, who is an emacs user, wanted to edit an important file.

Because emacs is too much to type he had setup an alias for emacs:

alias em=emacs

Under the influence of not enough or too much coffee he of course mis-typed em ...

Well, this is just another reason to use vi ... ;)

answered Jan 21 '11 at 21:09

maxschlepzig

57,532

What I really love about this one is the fact that he was concerned with how long emacs was but didn't just use alias e=emacs. – Steven D Feb 07 '11 at 16:55
Apart from coffee, notice that 'e' and 'r' are not very distant near in the keyboard... – leonbloy May 15 '11 at 21:13

score 11 · Answer 15 · answered Aug 27 '10 at 21:43

11

Or another experience, how to feel really stupid in a few easy steps that don't seem all that stupid individually.

Step one: establish an account for the kid, in case he wants to use a Linux box. Give it a trivial password, since after all this is a home system and isn't exposed to the net.

Step two: allow time to elapse, so you don't remember step one.

Step three: open the SSH port in the firewall (actually the NAT on the router) in order to ssh in. After all, my accounts have pretty good passwords, and it isn't like there's anything tremendously valuable.

Step four: get notification from ISP that there's some sort of DOS activity going to a Swedish site. Assume it's probably the Windows boxes, and examine and harden them.

Step five: get notification from ISP that it's still going on. Ask for some detail, get IP address of Swedish site, fire up Wireshark, find which box the attack is coming from.

Step six: clean up Linux box, feeling stupid. Find the login came from a Romanian address. Remove accounts without good passwords.

answered Aug 27 '10 at 21:43

David Thornley

363

1

Been there. I once created an account with a login of test:test, intending to use it for only about five minutes. I forgot to delete it and got same results as you. Never again. I go all public-key auth now. – ATC Oct 09 '10 at 17:00
1

Yeah, I also use public/private keys. They can be a bit annoying sometimes, but a lot safer than simple passwords. – Denilson Sá Maia Nov 13 '10 at 22:48
4

When opening SSH to the internet, NEVER EVER allow password authentication. Only public/private key auth, that way you can't get caught off guard. – wazoox Jan 21 '11 at 14:47
@wazoox: otp is a good alternative too. – Jürgen A. Erhard Apr 12 '11 at 23:22
I use a non-standard, high-numbered port for ssh, forwarded from the router to the standard ssh port on my Linux and Mac OS X boxes. I never get any attempted logins. I have the port configured in a shortcut in ~/.ssh/config on my other machines so usually I don't even have to type it. – Neil Mayhew Jul 05 '11 at 03:28

score 11 · Answer 16 · answered Nov 16 '10 at 20:54

In the computer labs when I was in college, they had a screen saver that simulated a bunch of balls that would float back and forth. They pulled on each with simulated gravity.

Once, while I was messing around with the settings, it crashed with the error Error: force on balls too great

score 10 · Answer 17 · answered Sep 12 '10 at 23:48

I was once developing a device driver for Unix. It had a pointer problem and during testing it started to write off the end of an array in kernel memory. I was slow to spot this and didn't hit the reset button immediately. The driver had scribbled all over the disk buffer cache which was then flushed to disk before I hit reset. A lot of the blocks were inodes and directories, and I ended up with a totally trashed filesystem. I think 6000 orphaned files were put into lost+found before I gave up and reinstalled. Fortunately, this was only a test system, not my workstation with all my files on it.

Now doing that on your work system would have been really stupid ;-) — Jürgen A. Erhard, Apr 12 '11 at 23:23

score 8 · Answer 18 · answered Aug 16 '10 at 05:44

8

I deleted /etc and then recovered it. I don't think I learned my lesson... I've had to recover from a deleted /bin too. Seems to happen when I've been working with a chroot.

answered Aug 16 '10 at 05:44

xenoterracide

59,188
74
187
252

I would very much like to know why I got downmodded – xenoterracide Aug 17 '10 at 01:01
1

I've been witness to this as well. I'm amazed that a system can survive without /etc --- just don't restart anything. – Stefan Lasiewski Aug 19 '10 at 17:06

score 7 · Answer 19 · edited Jan 14 '12 at 03:56

7

This happened to me last year. I was removing some files from the server using a temporary variable:

rm -rf ${prefix}*

Guess what? The variable $prefix was not defined!
You can imagine the disaster... it resulted in some very critical files deleted.

I almost broke the Control-C and ran to the CPU to remove the network cable!!

Hahaha I'm sure someone had already done this...

edited Jan 14 '12 at 03:56

Kevin

40,767

answered Oct 09 '10 at 13:06

jyz

667

score 7 · Answer 20 · answered Jan 20 '11 at 21:29

Last year, a colleague of mine was using one of our linux workstations to create copies of flash disks using the dd command. He accidentally typed something similar to the following:

dd if=flash-image.img of=/dev/sda1

By the time he realized his mistake - overwriting the machine's hard disk instead of the flash drive - the machine was already hosed. We had to rebuild the box, which incidentally was also the machine hosting all of our development VM's at the time...

+1 -- be very very very careful with dd!!! :-) – Josh Jan 27 '11 at 13:31 — Josh, Jan 27 '11 at 13:31

score 7 · Answer 21 · edited Jan 14 '12 at 04:10

7

While in my 2nd year of studying computer science we were given a homework assignment to write a program in C that would spawn a number of subprocesses with fork and make them communicate with pipes in a "circle" and figure out which one should be the "leader".

We were still quite noobs back then and most of peple didn't have any Linux machines, so we worked on our accounts on our faculty's main server (which was hosting official site and staff accounts and sites as well). Most of the people wrote forkbombs at some stage of trying to do the homework. Over half of my group got to the abusers file. That was the highest load on that server in a looong time :)

edited Jan 14 '12 at 04:10

Kevin

40,767

answered Aug 19 '10 at 09:08

naugtur

2,403

that was silly of them... they shouldn't have put you on the same machine as webservers and stuff. (My school has the same thing, but we have severs dedicated to remote shells so the yearly forkbombs don't really get too out of hand) – Spudd86 Feb 09 '11 at 03:26
Microsoft got the faculty to get into partnership and windows and .net were much promoted then already. Nobody expected a bunch of noobs on the server :) – naugtur Feb 10 '11 at 17:21

score 6 · Answer 22 · edited Jan 14 '12 at 04:13

6

I was a lab assistant for a Linux class. One of the students called me over because she could no longer su - because she was getting permission denied. OK, she's misremembered/mistyped the password. Reboot into single-user mode and reset. What?! su STILL doesn't work?! It MUST bow to my will! So I reboot into single user mode to find out what she did. I realized that she ran chmod -R 777 /var/www/html/drupal-6.19 /

Note the space between the directory name and the final slash.

After a few minutes of "I really don't want to have her reinstall, so what is this doing and how.", I managed to find that /bin/su now had file permissions of 777. That can also be read as file permissions of 0777, which removes the setuid bit from /bin/su. A quick chmod u+s /bin/su and I was a hero.

edited Jan 14 '12 at 04:13

Kevin

40,767

answered Oct 09 '10 at 22:26

Kevin M

1,973

5

That was the only thing you needed to change to unbreak her system? – Michael Mrozek Oct 10 '10 at 00:10
1

I'm sure it was still broken, but maybe it worked well enough for the rest of the class. – Stefan Lasiewski Oct 10 '10 at 03:53
1

It did work well enough for the rest of the class; the school evidently has a requirement that all student drives be reinstalled every quarter. And she wasn't going on with Linux after this quarter was over, so I didn't bother fixing it the rest of the way. – Kevin M Oct 10 '10 at 12:43

score 6 · Answer 23 · answered Jan 20 '11 at 20:35

6

As root on Solaris,

$ kill -9 1

...and everything went down.

My friend got fired because of this.

answered Jan 20 '11 at 20:35

lamcro

893

score 6 · Answer 24 · answered Aug 18 '10 at 13:19

When my University decided to switch the wireless network to use proprietary Cisco LEAP authentication...

Started a very long battle that ended well enough. Wrote up documentation for others who wanted to run Linux and have access to the internet. Six months later they decided to add PEAP support as well. face slap

It is my favorite because I won. I got it to work.

score 5 · Answer 25 · edited Aug 30 '11 at 19:13

5

A company that I used to work for had its product running on SCO. I was doing some debugging about applications getting very slow on our demo server and at the same time there was a bunch of customers being given a demo/lecture about upcoming new features.

So, I ran the application that used to get stuck, did my stuff on it to verify the root cause, but since it was still "stuck", I tried to kill it:

pkill -9 mytestapplication

What I did learn was that pkill doesn't do exactly the same on SCO as it does on linux =)

... It basically kills everything the user has access to, and with root... that's everything =)

edited Aug 30 '11 at 19:13

Jason Plank

105

answered Sep 02 '10 at 18:02

rasjani

101

1

Could you please describe what it does on SCO? I have no idea, and I couldn't find it easily on google. – Denilson Sá Maia Nov 13 '10 at 22:55
1

it basicly kills everything the user has access to, and with root .. its everything =) – rasjani Nov 18 '10 at 12:09

score 5 · Answer 26 · answered Aug 17 '10 at 23:43

5

Not that painful... But a fun little moment:

I've mistyped ls as sl and found out that the sysadmin had something installed for such case.

(already available in Debian, Ubuntu, Gentoo,... repositories)

answered Aug 17 '10 at 23:43

Denilson Sá Maia

1,821

4

That's more of an Easter Egg than a painful moment. I read about it before I came across it, but I've never found myself mistyping that, so no +1. – Umang Aug 19 '10 at 11:56
Grinn.. I had a user once that came to me "You're system is hacked!!!11one I got an ASCII-art animation when I typed ls" – blauwblaatje Jul 27 '11 at 09:56
^ Not so funny! – Vishwanath Dalvi Jul 31 '11 at 16:34

score 5 · Answer 27 · answered Mar 10 '11 at 10:02

5

git init
git clean -f

This does not remove the repository. This removes everything that is not in the repository.

After trying to get rid of the existing repo and then start source control again (on the completed first version of a project), these two commands nuked my entire code.

answered Mar 10 '11 at 10:02

Ankit

101

git rebase (with squashing) and git filter-branch (for removing files) are your friends :-) By this I mean that there are safer ways to remove cruft from your repo history and end up with a relatively clean project. I usually make a backup copy of the current branch before extensive rebasing. – Neil Mayhew Jul 05 '11 at 05:14

score 3 · Answer 28 · edited Jan 14 '12 at 04:19

On a Debian installation back in 1999. 14 floppy disks for the basic installation.

I tried to get xfree86 working. But X didn't start. I had to find out the settings of my graphics card (memory, horizontal and vertical refresh rate), which was completely undocumented. It turned out superprobe found out the correct amount of internal graphics card memory (1024 kB).

But it took me nearly a week to find out that the resolution setting (1024x768) didn't work. I had to switch it to 640x480 until the graphics card finally worked (at 1024x768... buuuuuug....).

I tried to get the serial port mouse to work on COM1. So I tried to get the mouse to work. Reading a book (back then I had no usable high-speed internet), I tried with

/dev/ttys0

And it didn't work and didn't work. It took me nearly another week to find out this was because I needed to type the S is uppercase, not lowercase...

/dev/ttyS0

It was about then when I finally realized what 'case-sensitive' really means.

score 3 · Answer 29 · answered Nov 16 '10 at 15:23

3

My switch from Debian to Ubuntu started the day I tried to delete some files and directories, meaning to type

rm -r /var/tmp/*

Unfortunately, I inserted a space between "/var/tmp/" and "*" and even worse, I was in the root of the filesystem.

root@workstation:/# rm -r /var/tmp/ *

Please don't try that at home!

answered Nov 16 '10 at 15:23

ddeimeke

4,597

'zsh: sure you want to delete all the files in / [yn]?' - it's quite useful – André Paramés Jan 21 '11 at 00:04
this will delete the files from / how come ? – Vishwanath Dalvi Jul 31 '11 at 15:24

score 3 · Answer 30 · answered Jan 21 '11 at 10:50

3

There's a neat trick to do the equivalent of dirname and basename, respectively, in Bash:

${path%/*}
${path##*/}

Not so funny when $path contains a trailing slash...

answered Jan 21 '11 at 10:50

l0b0

51,350

score 3 · Answer 31 · edited Jan 14 '12 at 04:16

I had two drives installed at one point and had the root filesystem of the second drive mounted in a directory within /mnt. I was in that directory and tried to delete var but ended up typing rm -rf /var instead. Some instinct seemed to kick in that said var must be preceded with a slash!

When I realised what I'd done I immediately hit Ctrl-C but it was too late. My rpm database had long since left the building. I spent ages getting everything back to normal.

Now for the painful part.

I go back into that directory in /mnt to resume what I'd been doing. What do I type? Well, let's just say that instinct kicked in again.

At least I was able to restore the system a lot quicker the second time ;)

I take it this was before you found out about chroot(1)? – Kevin M Oct 09 '10 at 22:13 — Kevin M, Oct 09 '10 at 22:13

score 2 · Answer 32 · answered Oct 09 '10 at 21:18

2

Mine was chmod -R 777 /: after that I couldn't figure out how to restore the permission on the whole filesystem and then I reinstalled the OS.

Never did that again (and still don't know how to restore such a situation).

answered Oct 09 '10 at 21:18

dag729

241

3

While you posted this, I was typing up how I recovered from just such a scenario: http://unix.stackexchange.com/questions/502/what-are-your-favorite-painful-unix-moments/2939#2939 – Kevin M Oct 09 '10 at 22:27
@kevin : Were you able to restore the rest of the system? I would think that would be possible, but painstakingly tedious. – Stefan Lasiewski Oct 10 '10 at 03:54
1

@Stefan Lasiewski: I'm sure I could have fixed it completely with a combination of mounting another student's root, andthen a combination of find, sed, and chmod --reference, but I didn't bother. – Kevin M Oct 10 '10 at 12:46

score 2 · Answer 33 · edited Jan 14 '12 at 04:31

First one

One time, I had defined an alias to help me clean temp files:

alias clean_dir_tmp="find /tmp -maxdepth 1 -user **** | grep *** | xargs rm -rf"

and of course one day, when I had forget what the alias was doing I mistyped:

$ clean_dir_tmp *

I think I lost a good couple of seconds before I realized what was happening... :(

Second one

I was working as usual with my Mac laptop and my Ubuntu desktop. When I plugged the external HD of the Mac (HFS+ filesystem) in Ubuntu, I noticed that the owner was ?????. My UID was different on my Mac and on my desktop so since I wanted a nicer output with ls -l, I decided to change that.

So I modified the UID on the mac and launched a big chown -R * on the HD from Ubuntu. The only thing that I didn't know was that the HFS+ driver for Linux was not stable. To this day, I haven't been able to mount this HD on Linux or my Mac again...

I've been burned by aliases, too, so nowadays I use bash functions instead, which are a much better-designed feature. — Neil Mayhew, Jul 05 '11 at 05:20

score 2 · Answer 34 · edited Jan 14 '12 at 04:21

2

I mapped the CapsLock to ESC on the entire system. When I did it, the Capslock was on.

A reboot removed the permanent state of CapsLock. It was mapped to ESC

It wasn't really painful but I felt stupid when I realized what I'd done!

edited Jan 14 '12 at 04:21

Kevin

40,767

answered Nov 16 '10 at 15:15

Luc M

4,095
5
30
29

score 2 · Answer 35 · answered Aug 16 '10 at 20:53

A long time ago, I was installing MkLinux on my Mac, and I wanted to replace the file that governed command processing (not the shell, something more basic, don't remember quite what anymore). The instructions said to do mv x y, so I decided to be cautious and started with rm y. The intention was to mv x y afterwards, but of course that didn't work. I reinstalled.

score 2 · Answer 36 · edited Jan 14 '12 at 04:22

I don't keep track of errors on my own boxes, but from the last 15 years here are my two work fatalities:

1995: Standard 'rm -rf as root' failure on a single box (not my design!) acting as a combined NIS master server + sole DNS server for the company + primary SMTP/POP3 server for the company. It was SunOS 4.1.3_U1 as I recall. Unsurprisingly, like the witness to a horiffic crime, I do not remember the ensuing 48 hours.

1998: Ran newfs (SunOS) on a production AFS (now OpenAFS) file server instead of the replacement we were standing up. Spent the afternoon and night restoring from tape.

score 2 · Answer 37 · answered Aug 20 '10 at 13:05

2

chown nobody:nobody /*

As root of course....

answered Aug 20 '10 at 13:05

Toby

3,993

may I ask... why? – Tobias Kienzler Jan 26 '11 at 17:31
Because I needed to set all to it in the current directory, not from the root directory up... – Toby Jan 27 '11 at 16:52

score 2 · Answer 38 · edited Jan 27 '12 at 21:09

Debian dist-upgrade to unstable (or was it testing?) on a remote production server.

Ignored warning about libc switching to nptl threads.

Not sure where installation actually failed, but I was left with one root console on a dial-up line with every single app exploding. Just one running ssh and bash.

Had a lot of fun of recovering it. Uploaded statically linked dpkg, rolled back libc, built a custom kernel with RAID support. I think that took around 3 hours with my dial-up line. When I finally rebooted it, fsck took like 10 minutes. Quite painful 10 minutes I should say.

score 1 · Answer 39 · edited Jan 14 '12 at 04:27

1

Back in the day you had to do erase the first 512 bytes of a partition to properly format FAT drives from Linux. This is done using the dd command.

dd if=/dev/zero of=/dev/hda1 bs=512 count=1

Except the FAT partition was /dev/hdb1

I didn't realize what had happened until after I rebooted. Luckily I was able to recover it by re-installing Lilo, or something.

edited Jan 14 '12 at 04:27

Kevin

40,767

answered Sep 02 '10 at 18:22

Mr. Shiny and New 安宇

163

score 1 · Answer 40 · edited Jan 14 '12 at 04:28

Twenty minutes ago, I was painstakingly recreating a complex directory structure from files I had on my other PC. I decided to run du to see if it was near completion. Given the size, I knew it would take few minutes, so went to get some coffee.

On my return I noticed, to my severe dismay, that instead of running

du -hs /path/to/important/folder

I had absent-mindedly run

rm -r  /path/to/important/folder

score 1 · Answer 41 · edited Jan 14 '12 at 04:39

I wanted to delete a file and its backup copy (file and file~) on a SuSE system I rarely used and didn't notice that command completion was configured differently. I typed rm fi<tab> and expected the cursor to be after the last letter. The system ignored ~ files and inserted a space after the file name.

So I wanted to type

rm file*

and I did

rm file *

Now I am always checking what the suggested completion is.

score 1 · Answer 42 · edited Jan 14 '12 at 04:34

The first time I installed GNU Linux on my desktop, I installed Debian, no help, I only installed the basic system, no GUI. And I was like:

"OMFG OMFG WHAT DO I HAVE TO DO NOW?... I will need to go back to Windows"

But then I remembered how to install packages (first time in GNU Linux, only a jose@debian:$ output and only read some things from Debian) and the name of an IRC Client: IRSSI and the name: GNOME and then I installed them... since that day, I install IRSSI and GNOME in every machine...

It feels... good experimenting with your computer xD

score 1 · Answer 43 · answered Nov 17 '10 at 16:44

1

Somehow managed to unmount /dev and thought I would be screwed forever if I rebooted the machine.

Nerve wracking hour ensued trying to figure out if it would be safe to reboot it. It was, nothing bad happened.

answered Nov 17 '10 at 16:44

mattis

83

score 1 · Answer 44 · edited Jan 14 '12 at 04:33

Many web apps like magento or eZ Publish have a var/cache/ folder and a way to clear the cache that is faster than going in the app backend is to do this:

rm -rf var/cache

After doing this a couple of times, it is scientifically proven that you always end up:

Either doing this in the root of your server; or
Adding a / before var

Lesson learned: create an alias for this command with an absolute path in it.

score 1 · Answer 45 · edited Jan 14 '12 at 04:23

1

Trying to get the Xwindows driver for my Nvidia card working when Fedora initially released the Nouveau driver. I had downloaded the Nvidia source to compile and install myself as I had many times in the past, but this release, I could just not get it to work. There were quite a few steps to find in the Fedora Forums to completely disable the Nouveau driver, and get the Nvidia driver working. Quite painful to say the least.

edited Jan 14 '12 at 04:23

Kevin

40,767

answered Aug 15 '10 at 01:36

user239

56

Nvidia makes my brain scream. I only ever use intel GPUs these days. – Eli Frey Aug 15 '10 at 04:16

score 1 · Answer 46 · answered Feb 07 '11 at 14:28

1

Wanted to make an archive:

tar -cvzf mydir/* mydir.tar.gz

Of course, mydir/* expanded to mydir/myfile.cpp mydir/myfile.h

Remember that the archive name follows the -f option of tar!

answered Feb 07 '11 at 14:28

Benoit

381

score 1 · Answer 47 · answered Feb 07 '11 at 16:30

1

My favorite is when I was building the new Solaris system for an Oracle database installation. Everything was in place including the high priced oracle consultant right in the middle of the DB optimization work he was doing. I was in the server room checking on another server, when I tripped over the power cord to the server.

answered Feb 07 '11 at 16:30

you forgot to put how many hours were wasted with that trip. :) – lamcro Oct 05 '11 at 21:21

score 1 · Answer 48 · edited Jan 14 '12 at 04:36

1

1st ex employee: How do I do (something trivial)?
2nd ex employee: sudo rm -rf /
1st ex employee: Haha ok
...
1st ex employee (Having forgotten that he'd just sudoed something else, so it didn't ask for his password again) : oh F**K!!!

edited Jan 14 '12 at 04:36

Kevin

40,767

answered Feb 07 '11 at 23:02

score 1 · Answer 49 · answered Mar 09 '11 at 13:33

1

I was once trying to zero out a USB thumbdrive using dd.

Needless to say, when tty1 started spitting out ReiserFS errors from my root partition, I had to reinstall...

answered Mar 09 '11 at 13:33

new123456

585

score 1 · Answer 50 · edited Jan 14 '12 at 04:37

1

Installing Debian (with net-install) on a computer and realise after rebooting that I had somehow skipped the part where one is supposed to choose which packages to install.

Sure, a non-graphical system with basically only pwd, ls and cd is working just fine ;).

edited Jan 14 '12 at 04:37

Kevin

40,767

answered Mar 10 '11 at 12:57

Alex

233

huh? Debian net-install gives you more than enough. Most servers I install don't get much more during the basic install. Who needs a graphical system anyway :) – blauwblaatje Jul 28 '11 at 07:41

score 0 · Answer 51 · edited Jan 14 '12 at 04:38

0

I did chown -R /usr. Somehow, I thought it was a good idea. Subsequently, a puppet script did some bad stuff, and somehow I lost sudo rights.

Our qualified sysadmin wasn't able to recover my sudo rights, and I had to re-image the machine.

edited Jan 14 '12 at 04:38

Kevin

40,767

answered Apr 12 '11 at 06:31

ripper234

31,763

score 0 · Answer 52 · answered Feb 08 '11 at 21:30

A while ago, I needed to do some extensive configuration on one of my machines, that mostly involved editing bunch of files in /etc. I decided to be really careful about it, so I created an etc directory in my $HOME, copied the files that needed editing in there, spent couple hours doing the edits. I carefully checked all the files, made sure that all the edits were exactly the way they were supposed to be, logged in as root and copied the edited files back to /etc. By then, it was very late at night. Still as root, I decided to clean up, and instead of rm -rf etc, I typed in rm -rf /etc. I did not get much sleep that night.

score 0 · Answer 53 · answered Feb 09 '11 at 03:23

I was on the phone with a colleague who was out at a customer site. She was working on their systems and I was telling her stroke for stroke what to type, she misheard me when I told her rm -rf .??* and typed rm -rf .?*. But she wasn't in the directory she had told me, she was in the root directory. Wiped not only the dot files, but the entire OS.

Back when I was "learning as a sysadmin", I was writing my own adduser script (didn't have it on early SysV). A shell error in the script (cat /etc/passwd; echo ...) > /etc/passwd which of course wiped the passwd file and then, by accident, I hit Ctrl-D to exit the su shell. Had to go in to the office at 1am Saturday morning to get the boot diskettes.

What are your favorite painful Unix moments

53 Answers53

Linked

What are your favorite *painful* Unix moments

53 Answers53

Linked

What are your favorite painful Unix moments