8

I mistakenly have source .bash_profile in the bashrc file and vice versa. Now when I tried to ssh into the machine (ec2), it will stuck at loading bash and get connection closed in a second. Is there a way I could fix it? Could I mount the disk to another ec2 instance to fix the bash files?

Update 1: I tried the following:

%  ssh -i "my-pem.pem" -t ubuntu@<server_address>.amazonaws.com "/bin/bash --noprofile --norc"
Connection to <server_address>.amazonaws.com closed.

Nothing else showed up. Do you have any idea on what was going wrong?

For sanity check, if I do ssh -i "my-pem.pem"ubuntu@<server_address>.amazonaws.com, the message will be

...

28 packages can be updated.
0 of these updates are security updates.
To see these additional updates run: apt list --upgradable


New release '20.04.2 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


Last login: Mon Feb 22 23:17:41 2021 from ip
Connection to <server_address>.amazonaws.com closed.

Solution

Just ssh onto the machine and immediately do Ctrlc and fix the bash files.

Quasímodo
  • 18,865
  • 4
  • 36
  • 73
John M.
  • 181
  • 11
    You should add your solution as an answer, not as part of the OP. This might help someone out in the future. – ajgringo619 Feb 23 '21 at 03:08

3 Answers3

6

In general, ~/.bashrc is only sourced by interactive shells, and ~/.bash_profile by login shells (Difference between login shell and non-login shell?). If you only send commands via Ssh, you get neither an interactive nor a login shell, so just renaming one of the files—say, ssh user@host 'mv .bashrc .bashrc.bak'—would work.

But there is an exception, as Xhienne points out: If invoked by the SSH daemon, Bash does source Bashrc.

However, Xhienne and I failed to reproduce that. My test was to append touch testfile to the server's Bashrc and then try a dummy command to see what happened: ssh Quasímodo@server 'echo x'. Result: No testfile was created. Only later did I find out that my Bashrc contains as its first lines

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

I.e., if the shell is not interactive, stop reading Bashrc. This nifty trick is actually present in most distributions as thoroughly discussed in Why does bashrc check whether the current shell is interactive?

Conclusion

Andy's, Xhienne's and my original answers, that amounted to running a command through SSH, will

  • Succeed if that fragment is present in the user's remote Bashrc.
  • Fail otherwise, in which case you can find some options at Server Fault, among which Ctrl-C.
Quasímodo
  • 18,865
  • 4
  • 36
  • 73
  • That won't work, because the ssh daemon actually uses your shell (in interactive mode) to run the passed command. – Toby Speight Feb 23 '21 at 17:36
  • One would also need to create an account with Bash as login shell! The question says bashrc though - did that really mean ~/.bashrc rather than /etc/bashrc? It's absolutely unclear what should be tested. My comment was based on the documentation quoted in xhienne's answer. – Toby Speight Feb 24 '21 at 12:07
  • 1
    @TobySpeight Updated answer: I have tracked down the behavior to some lines in Bashrc itself. – Quasímodo Feb 25 '21 at 21:14
  • 1
    That's great, when all the different approaches come together. And I was wrong to say "(in interactive mode)" because Bash doesn't set -i in that case, even though it sources the rc file. – Toby Speight Feb 26 '21 at 07:35
6

[update]

This interesting case deserves a better answer (which I'm borrowing from this answer on superuser).

My original answer below depends on the fact that ssh user@server command is supposed to execute command in a non-interactive shell, thus keeping the fault .bashrc from being executed.

The problem lies with the remote shell. If it is Bash, then there is a special features that makes it execute those rc files anyway:

(man bash) Bash attempts to determine when it is being run with its standard input connected to a network connection, as when executed by the remote shell daemon, usually rshd, or the secure shell daemon sshd. If bash determines it is being run in this fashion, it reads and executes commands from ~/.bashrc, if that file exists and is readable. It will not do this if invoked as sh. The --norc option may be used to inhibit this behavior, and the --rcfile option may be used to force another file to be read, but neither rshd nor sshd generally invoke the shell with those options or allow them to be specified.

That means that if your remote shell is Bash, then even if you run ssh user@server 'rm .bashrc', your .bashrc will be still executed first and your never-ending loop will prevent the rm command from being executed.

Note that scp won't save you since it is also a command executed internally by sshd by calling sh -c scp ....

The only exception is SFTP. sshd has an SFTP server built-in. If sshd is configured to use this internal SFTP server instead of an external one, then the server won't be started with bash -c ....

So, if your remote default shell is Bash, your only option is to use the "Download-edit-upload" method below, not with scp but with sftp.

[original answer]

Depending on your installation, here are several options:

Edit your .bashrc file:

ssh -t user@server 'vi ~/.bashrc'

Run bash without your .bashrc file:

ssh -t user@server '/bin/bash --noprofile --norc'

Run another shell:

ssh -t user@server /bin/sh

(sh is an example; you may try any other shell that is available on your server, like csh or fish)

Download-edit-upload you faulty .bashrc:

$ scp user@server:~/.bashrc /tmp/bashrc
$ vi /tmp/bashrc
$ scp /tmp/bashrc user@server:~/.bashrc
xhienne
  • 17,793
  • 2
  • 53
  • 69
  • All of these returned a connection closed message like the one in the updated question. Any thoughts on what was going on are much appreciated. – John M. Feb 22 '21 at 23:21
  • @JohnM. I have added some ideas but your feedback makes me wonder if there is another problem than the one described in your question (typically you might be denied access because of two many ssh connections in a short period of time). I'm also wondering if your ssh connection allows you to execute anything else than the shell that was configured there (in the past, did you use ssh u@server command or scp?) – xhienne Feb 22 '21 at 23:32
  • There is as far as I know no (at least not sufficiently low) limit of sshing to that server. I tried your new command and % scp -i "my-pem.pem" ubuntu@e<server_address>.amazonaws.com:~/.bashrc /tmp/bashrc % ls -al /tmp/bashrc ls: /tmp/bashrc: No such file or directory but it doesn't seem to work :( – John M. Feb 22 '21 at 23:36
  • This is strange you have to use ls to see if it worked or not. Didn't scp output any error??? – xhienne Feb 22 '21 at 23:38
  • I used ls above (apologize for the formatting) and there is nothing. – John M. Feb 22 '21 at 23:40
  • This is not my question: I'm asking whether scp yielded any error. I also would like to know if you had already used scp and ssh u@server command with that server in the past. – xhienne Feb 22 '21 at 23:44
  • Ah I used scp and it worked smoothly in the past before I tweaked the bash files. Haven't used ssh u@server command in the past. It is a regular EC2 instance and nothing is really special IMO. – John M. Feb 22 '21 at 23:48
  • Good to know, thanks. I'm also asking whether the scp user@server:~/.bashrc /tmp/bashrc command yielded any error. – xhienne Feb 22 '21 at 23:50
  • No error but the file wasn't scp-ed successfully. The above message is the full log. – John M. Feb 22 '21 at 23:53
  • Let us continue this discussion in chat. – xhienne Feb 22 '21 at 23:54
  • 1
    I'm not sure about the updated answer. I can't reproduce it on an OpenSSH server I have at hand: adding echo X > test to the server's .bashrc and then trying ssh user@remote 'mv .bashrc .bashrc.bak' works alright, no test file pops up. I think there might be some misreading of that part of the manual. – Quasímodo Feb 23 '21 at 12:47
  • @Quasímodo I do agree with you, I can't reproduce the behavior on my server either. I assume this might be the case with some older versions of Bash. Anyway, the sftp solution should work in all cases. OP solved his problem by ctrl-C the running script, so this confirms that something is executed despite the non-interactive shell. – xhienne Feb 23 '21 at 12:52
  • 1
    Dropping by to say that I think I've found the missing link in case you want to update your answer: Most .bashrcs switches off if the shell is non-interactive. Do you also verify this in your server? – Quasímodo Feb 25 '21 at 21:43
  • 1
    @Quasímodo Well spotted! On my Debian server, I have [ -z "$PS1" ] && return in my /etc/bash.bashrc and something similar in /etc/profile. That indeed explains why I'm not affected by bash's behavior. Whereas I should modify my answer, I lack the time right now, but I'm not sure there is anything wrong in the updated part: this explains the OP's issue and provides an acceptable solution. If you see anything to change, don't hesitate to edit it. I'll come back tomorrow and review it more thoroughly anyway. – xhienne Feb 25 '21 at 21:59
6

When you run ssh, by default it runs your shell in interactive mode, and in interactive mode it processes your .bashrc. However, if you specify a command to run, it will invoke the shell in non-interactive mode (so it won't process your .bashrc).

You can use that to launch an interactive shell, but specifying command-line arguments to that shell to instruct it not to process your .bashrc:

$ ssh -t user@host "/bin/bash --noprofile --norc"
bash-5.0$

If the problem is in your .bashrc file, this may help you to log in. However, if the problem is in a bash-related profile file (e.g., .bash_profile), then you may still get hung up in the first non-interactive shell.

Andy Dalton
  • 13,993
  • 1
    Trouble is, it passes your command to a shell on the far end, rather than directly invoking it. I don't think there's a way to avoid that. – Toby Speight Feb 23 '21 at 17:35
  • 1
    @TobySpeight However, it seems that when called like this, the bash called implicitly (the bash that's calling bash here) is neither a login shell nor interactive, so neither the profile nor rc files are sourced. --noprofile is redundant for the child bash. – JoL Feb 23 '21 at 20:35
  • 1
    I may be nitpicking word-choice but, "When you run ssh, by default it runs your shell." -- That's always the case, I believe. It's not a matter of defaults. "You can, however, specify a different command for it to run." -- You can specify a command for your shell to run, not in-place of the shell. – JoL Feb 23 '21 at 20:55
  • @JoL yup, you're right. I mistakenly thought that it exec-ed the given command, but as you point out, it execs a shell that runs the given command. – Andy Dalton Feb 25 '21 at 01:26