Different behavior between SSH into VM and running SSH commands from script

Question

I'm relatively new to shell scripting, so I apologize if this seems like a simple question to ask. I have a linux VM (debian, version 11 (bullseye)) that I can ssh into (ssh <ip>), installed a few dependencies (homebrew, pyenv, etc) and was able to successfully use them. However, when I try running commands from outside of the VM (ssh <user>@<ip> pyenv versions) in either scripts or using my Mac terminal, I get bash: line 1: pyenv: command not found related errors.

I think this may have something to do with whats explained here, but I'm not entirely sure how to circumvent that problem.

Adding additional details asked by @terdon in comment below:

$ which pyenv
/home/linuxbrew/.linuxbrew/bin/pyenv
$ grep /home/linuxbrew/.linuxbrew/bin/ ~/.bashrc ~/.bash_profile ~/.profile /etc/bash.bashrc /etc/profile
grep: /home/f0p021s/.bash_profile: No such file or directory
/home/f0p021s/.profile:eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"

I've also realized that if I look at my path from within my VM it looks like this:

$ echo $PATH
/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

And when I try to run a similar command from my local machine it looks different:

$ ssh <user>@<ip> 'echo $PATH'
/usr/local/bin:/usr/bin:/bin:/usr/games

score 11 · Accepted Answer · answered May 19 '22 at 16:28

When you ssh into a machine, you start an interactive login shell. When you run ssh ip command, you are starting a non-interactive, non-login shell:

$ ssh localhost 'echo $- $0; shopt login_shell'
hBc bash
login_shell     off
$ ssh localhost
[terdon@tpad ~]$ echo $- $0; shopt login_shell
himBHs -bash
login_shell     on

_{See this answer for details on what this is actually showing you.}

The files read on startup by each type of shell are different. From man bash (emphasis mine):

When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The --noprofile option may be used when the shell is started to inhibit this behavior.

When bash is started non-interactively, to run a shell script, for ex‐ ample, it looks for the variable BASH_ENV in the environment, expands its value if it appears there, and uses the expanded value as the name of a file to read and execute. Bash behaves as if the following com‐ mand were executed:
    if [ -n "$BASH_ENV" ]; then . "$BASH_ENV"; fi
but the value of the PATH variable is not used to search for the file‐ name.

Now, you have shown us that the pyenv command is added to your $PATH in /home/f0p021s/.profile. As you can see above, that file (~/.profile) is read by interactive login shells but not by non-interactive or non-login shells as those only read whatever is pointed to by $BASH_ENV and that is empty by default.

So, your options are:

Just use the full path to the command:

ssh ip /home/linuxbrew/.linuxbrew/bin/pyenv

Source ~/.profile:
```
ssh ip '. ~/.profile; pyenv'
```

LSerni · Answer 2 · 2022-05-20T12:30:04.723

Your diagnosis is probably correct.

To circumvent the problem, you need to verify which rcfile is not executed at startup (it will likely be the one where PATH is set up), among the scripts in the answer you refer to.

Then, you could rewrite your ~/.bashrc to avoid the problem.

Or you could try and change the command:

ssh <user>@<ip> '. /etc/profile; pyenv versions'.

This will import /etc/profile into the current executing shell. You might need to import /etc/bashrc or ~/.profile instead of /etc/profile, or dot them both (note there is a space between the dot and the file name; also, the whole command is quoted).

Testing (on Ubuntu)

Logging in on a handy virtual machine:

leonardo@aladdin:~$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/android-sdk-linux/platform-tools:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin
leonardo@aladdin:~$ logout

Now trying with straight SSH:

$ ssh leonardo@aladdin "echo \$PATH"
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

Okay. So, different PATHs. Where does, say, that java-8-oracle come from? You will do some similar check, with some lengthy string in the PATH that is only there when in a login shell.

As root from the VM, I look for that string following a PATH assignation, everywhere in /etc (should it not work, I'll retry with the home dotfiles by specifying /home/.*).

# grep -r PATH.*java-8-oracle /etc
/etc/profile.d/jdk.csh:setenv PATH ${PATH}:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin
/etc/profile.d/jdk.sh:export PATH=$PATH:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin

The PATH assignment is there, plain to see.

And it comes from a file in "/etc/profile.d". Now a small bit of trivia is needed - knowing that the "whatever.d" folders are handy folders usually parsed from a whatever binary, and their contents is globbed sequentially.

So if the folder is /etc/profile.d, I expect this trick to be turned out by /etc/profile. Anyway, someone must meddle with "profile.d", so I can look for that:

# grep -r "profile\.d" /etc

And indeed I find, among several lines of output:

/etc/bash_completion.d/libreoffice.sh:# It is based on /etc/profile.d/complete.bash from SUSE Linux 10.1
/etc/profile:if [ -d /etc/profile.d ]; then
/etc/profile:  for i in /etc/profile.d/*.sh; do
/etc/apparmor.d/snap/abstractions/bash:  /etc/profile.dos                 r,
...

The second and third row are what I'm interested in. They say, "If there is a folder called /etc/profile.d, then; for each file ending in ".sh" in that folder, do...". This is exactly what I would have expected.

So, I can be confident that running /etc/profile will do the trick.

And indeed it works:

ssh leonardo@aladdin ". /etc/profile; echo \$PATH"
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/android-sdk-linux/platform-tools:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin

I tried your second suggestion with both profile and bashrc and didnt work. Is there anything you would recommend to read to try and diagnose the rcfile approach? — Frank Pinto, May 19 '22 at 14:13
Add in each rcfile a line like 'echo Stating file`. Now login and compare the list of greeting lines with the expected list. — White Owl, May 19 '22 at 14:22
@FrankPinto some useful resources (some are my own, apologies for the self promotion, I just happen to know them): Difference between Login Shell and Non-Login Shell?, Difference between .bashrc and .bash_profile, Why are scripts in /etc/profile.d/ being ignored (system-wide bash aliases)?, Differentiate Interactive login and non-interactive non-login shell Sequence of scripts sourced upon login. — terdon, May 19 '22 at 16:33

Different behavior between SSH into VM and running SSH commands from script

2 Answers2

Testing (on Ubuntu)