1

This is rather strange story I want to share in case this originates from some kind of hack, so we can better identify signature of the attack.

I work as web developer, I run Ubuntu 20.04, I run strange things in docker on my machine, I have VSC with many development extensions of doubtful quality. I use ansible here and there Python venvs etc. I use ssh agent, my ssh key is password protected and I don't decrypt ssh key after login automatically.

Yesterday I was working as usual, switched off computer in the evening. Today I couldn't ssh to my remote machine with this message.

sign_and_send_pubkey: signing failed for RSA "/home/xxxxx/.ssh/id_rsa" from agent: agent refused operation
ubuntu@xxxxxxxx.xx: Permission denied (publickey).

After little fiddling I checked my private key and to my surprise it was destroyed.

$ cat ~/.ssh/id_rsa
-e 

-e is the content of the file, not happiest moments, but I have cold backup so I resumed work after a while.

But I can't sleep without knowing how this has happened. Lately I did not generate any new keys I did not manipulate files in .ssh/ manually.

The timestamp on the id_rsa is yesterday mid-day during my work, also known_hosts has the same timestamp, other files in .ssh/ have legitimate historical timestamps. Around the time I was sshing onto some old machines, so I can't see reason to touch known_hosts file.

Can anyone think of a situation which would lead to overwriting id_rsa with such non-sense as -e? I'm paranoically thinking of a malicious code in some of the tools I use. Or of course my human error, but I'm usually quite vigilant. Thanks for your ideas.


Edit:

  1. My bash_history is not complete, because of sub-optimal config of my bash and tmux, see https://askubuntu.com/questions/80371/bash-history-handling-with-multiple-terminals/80882. So tracking down specific command is unfortunately not posible.

  2. Based on answer I've discovered that VSCode creates lot of files starting with -, e.g. ~/.config/Code/User/History/-707c1648. Also I was lately having some performance issues with linters (running for couple seconds) and I've seen some temporary files being created next to the processed file. It lead me to hypothesis that maybe there is a clash in VSCode between git command and temporarily created file. This would need extra attention to track down.

VasekCh
  • 113
  • 2
    Does history | grep -E 'id_rsa|-e' shed any light? – Chris Davies Jul 26 '23 at 14:08
  • 1
    Good call @roaima There is nothing, I've also checked any ssh commands I've issued and there is nothing suspicious. Only ssh <name> using names from my .ssh/config – VasekCh Jul 26 '23 at 14:46
  • 2
    Out of caution I'd suggest generating a new key pair and removing your old public key from the servers. To me it looks like a "mistyped" command is the culprit here, but other than checking history there's nothing that comes to mind. – Panki Jul 26 '23 at 15:04
  • Yeah, it's good opportunity to refresh keys, I'm in process of replacing the old one. Thanks. – VasekCh Jul 26 '23 at 15:13
  • 2
    do you have the modification time of the file? that would help tie the update back to other system activity that could have caused this. I would check any shell scripts or shell aliases/functions that call rsync/git/ssh, as they may have some improper variable handling that could have caused this. -e seems like it was a shell script gone haywire, then any sort of malicious behavior. – toppk Jul 26 '23 at 16:36
  • 1
    I strongly suspect (based on 60+ years) $USER error. Losing track of context, which program $USER is running on which system (makes command history harder, but see fill in. Alternatively, Fumblefinger disease? Restore the file from backup. You do have backups, right? – waltinator Jul 26 '23 at 19:54
  • 1
    Replace "fill in" with https://askubuntu.com/questions/80371/bash-history-handling-with-multiple-terminals/80882 – waltinator Jul 26 '23 at 20:00

1 Answers1

3

No, it's not an attack. It's a mistake somewhere.

Replacing a private key file by invalid content is not a desirable goal for an attacker. An attacker could want to read a private key, but there's no point in changing a private key that's used for authentication, let alone destroy it by replacing it with some invalid text. This gives the attacker no advantage and is easily detectable. An attacker could want to overwrite known_hosts, but only to add their own public key: destroying your own would not give the attacker any advantage either. If it's an attack, it's one that failed in a very weird way, and somehow the attacker didn't even try to clean up. Sure, it's not against the laws of physics, but it is utterly implausible.

With -e as the content, it's very likely a mistake in some shell command where -e was intended as an option but was interpreted as content. With other files modified at the same time, a likely culprit is some file content replacement command: something like find … | xargs perl -i -p -e … where, due to bad quoting or a copy-paste error or a stray space or an unexpected character in a file name or something, the command ended up running on more files than intended.

Check if you have a file name starting with a dash somewhere (locate /- will find it if it's publicly visible, otherwise find ~ -name '-*'). Or perhaps a file name with a space followed by a dash (locate ' -' or find -name '* -*'). That's a likely culprit for mysterious effects resulting from a command that passes file names without using -- and, with the space, not using quotes around variable expansions or using xargs without -0. If you manage to find the culprit and it turns out to be in a shell command that you wrote, see Why does my shell script choke on whitespace or other special characters? and Security implications of forgetting to quote a variable in bash/POSIX shells. If it turns out to be in a shell command that someone else wrote, report a bug and point them to those resources. In your case it was accidental, but an attacker who can control the name of a file could exploit that bug to do useful things.

  • Thanks for the analysis, I can imagine things go wild by my error using STDOUT/STDERR redirect, tee command and similar and I get a content of a file replaced by something that was supposed to be an argument. But what I don't understand is why "~/.ssh/id_rsa" was the target file. This is not something I manually touch. Based on your ideas I found that VSCode creates history files like ~/.config/Code/User/History/-707c1648 and a lot of them. I elaborate on thin in update of my question. Marking this as correct answer, although it may not be final, thanks! – VasekCh Aug 02 '23 at 11:00