Have you ever had problems using ssh to run commands on a remote machine? Ever wondered, eg, why the PATH on the remote machine wasn’t getting set properly?
Some time ago (2007-ish) I ran into this problem while debugging svn+ssh URLs on Subversion. Since I no longer use Subversion, and because I buried the nuggets of gold that I found in an excessively prolix description of my methods of deduction, I’ve decided to rewrite this page to put the really useful bits at the top, and move the long, verbose version to the end.
I’m going to assume that you’re running Bash on the remote machine. All of the tests that I orginally did were to determine the behavior of Bash. sh, zsh, csh and others might be susceptible to the same analysis, but the details will surely differ.
Short version
To begin to debug PATH (and other environment variable) problems, do this:
ssh <remote> env | sort | less
This will run env on the remote machine and sort and display the results. Is the PATH correct? If not, you are probably setting it in your .bash_login or .profile on the remote machine, but this won’t work! Why? Because Bash doesn’t source those files when run by sshd. (To find out where this is “documented”, skip to the long version of this story, below.
An incredibly useful tool for debugging this is to set (and export) environment variables in the remote machine’s Bash startup files to see which files are actually getting sourced when Bash is run by sshd. So, put
export DOTBASHRC=1
into the remote machine’s .bashrc,
export DOTBASH_LOGIN=1
into .bash_login, and
export DOTPROFILE=1
into .profile.
Now try running
ssh <remote> env | sort | less
What did you get? When I did this, I got this:
DOTBASHRC=1
Moral: Set all of your important environment variables in the .bashrc file, and source it from .bash_login (or .profile, if you use that). This way the variables get set, no matter what.
That was easy, right?
Now read about the painful process I went through while trying to figure all this out!
Long (ago) version
Situation: I was setting up a Mac mini (OSX 10.4) to host Subversion repos, transitioning from FreeBSD.
In the commands that follow, osxbox represents the name of the OSX machine, and bsdbox the FreeBSD machine.
After building Subversion on OSX, and installing a separate (and newer) version (of Subversion), I wondered which one I would get using the svn+ssh scheme:
svn ls svn+ssh://osxbox/home/svn
This was the result:
bash: line 1: svnserve: command not found svn: Connection closed unexpectedly
svnserve is the protocol server that gets run on the remote end when you use svn+ssh.
Running
ssh osxbox env
yielded this:
PATH=/usr/bin:/bin:/usr/sbin:/sbin
The older Subversion was in /usr/local/bin; the newer one in /usr/local/subversion-1.4/bin. Neither was on this PATH.
I use Bash – both on the OSX server-to-be and on the FreeBSD box I’m ssh’ing from. I had to figure out how to debug which Bash startup files get sourced when logging in via ssh. I spent a long time reading the ssh, sshd, ssh_config, and sshd_config man pages, but they said nothing about shell startup files. I knew I didn’t have a .ssh/environment file – which is a way to set enviroment variables on the remote machine – and I wasn’t passing environment variables over the ssh link. So where were they getting set?
This used to work on FreeBSD. Somehow on FreeBSD Bash’s PATH was set to include /usr/local/bin, which is where the Subversion binaries – in particular svnserve – live. In contrast, on OSX, the PATH is rather minimal.
Why the difference? The Mac runs a newer version of sshd than my FreeBSD box. But the sshd man pages weren’t significantly different. I didn’t think that the sshd on FreeBSD was doing something special with the PATH.
After some digging, I found an interesting difference between FreeBSD and OSX: FreeBSD has login classes, and the standard login class sets a PATH that includes /usr/local/bin – which is where “add-on” software, such as Subversion, gets installed. OSX lacks login classes, so the PATH needs to be set by a shell startup script. On FreeBSD running
ssh bsdbox env
yields
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:\
/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:\
/home/david/bin
Conclusion: On OSX we need to set the PATH ourselves. The big question is: Where? When you ssh into a remote machine where you run Bash, which startup scripts get sourced? Any guesses? Here are a few possibilities:
/etc/profile $HOME/.profile $HOME/.bashrc $HOME/.bash_login $HOME/.bash_profile
I read the Bash man page (which is really confusing). Depending on whether the shell is a login shell, an interactive shell, or both, it sources different files. So what kind of Bash are we getting over ssh, when we request command execution rather than a login? (Remember, svn+ssh runs svnserve on the remote end, rather than running a login shell.)
I tried this (after first failing because I used double-quotes – sigh):
ssh osxbox 'echo $BASH $0 $*'
and it yielded
/bin/bash bash
which tells us the complete path to bash, and how bash was invoked. (Well, sort of. We can’t see bash’s options, but we can see its parameters – none, in this case.) ssh’ing into the FreeBSD box things are similar, but bash is in a different place:
ssh bsdbox 'echo $BASH $0 $*'
yields
/usr/local/bin/bash bash
When I first tried this (with the wrong quotes) I got a confusing answer:
ssh osxbox "echo $BASH $0 $*"
gives
/usr/local/bin/bash -bash
which is confusing for two reasons:
- -bash is a login shell, but we’re not logging in, and;
- /usr/local/bin/bash is where the local bash is (I ran this on bsdbox, sshing into osxbox)
and that’s how I knew my quotes were wrong. So now I knew how Bash was getting run. Here’s what the man page says:
A login shell is one whose first character of argument zero is a -, or one started with the --login option.
Our shell is “bash” and not “-bash”, but we don’t know if it’s being given -l or --login – we can’t see that. So it could be a login shell.
An interactive shell is one started without non-option arguments and without the -c option whose standard input and error are both connected to terminals (as determined by isatty(3)), or one started with the -i option. PS1 is set and $- includes i if bash is interactive, allowing a shell script or a startup file to test this state.
Again, we can’t tell what option arguments were given to bash, and how to tell if we are connected to terminals? What does $- contain?
ssh osxbox 'echo $BASH $0 $* $-'
gives
/bin/bash bash hBc
Hmm. No “i” in there, so bash thinks it’s not interactive. The “h” means “hashall” and the “B” means “brace expansion”. There is no mention of “c”, but I think it has to do with being passed commands via -c:
bash -c 'echo $BASH $0 $* $-'
yields
/bin/bash bash hBc
on OSX. We seem to be getting a non-interactive shell, but we can’t be sure it’s not a login shell. It probably isn’t. It shouldn’t be. And remember that if I run
echo "$BASH $0 $* $-"
in an interactive, login shell I get
/bin/bash -bash himBH
The “i” in the options means interactive; the “-bash” means login.
Aside: One reason, perhaps, for preferring csh over bash is that it’s trivially easy to make this distinction in csh: it sets the variable $loginsh for login shells.
Let’s assume our shell is a non-interactive, non-login shell. What scripts would it source? Here is what the man page says:
When bash is started non-interactively, to run a shell script, for
example, it looks for the variable BASH_ENV in the environment, expands
its value if it appears there, and uses the expanded value as the name
of a file to read and execute. Bash behaves as if the following com-
mand were executed:
if [ -n "$BASH_ENV" ]; then . "$BASH_ENV"; fi
but the value of the PATH variable is not used to search for the file
name.
BASH_ENV is unset, so nothing happens here. It would seem that it doesn’t source any files. Ah, but there is one last piece, a gross hack:
Bash attempts to determine when it is being run by the remote shell daemon, usually rshd. If bash determines it is being run by rshd, it reads and executes commands from ~/.bashrc, if that file exists and is readable. It will not do this if invoked as sh. The --norc option may be used to inhibit this behavior, and the --rcfile option may be used to force another file to be read, but rshd does not generally invoke the shell with those options or allow them to be specified.
Our shell is being started by sshd rather than rshd, but Bash might be able to figure this out.
In order to be absolutely sure of what was going on, I finally lit on a simple but powerful idea: set and export a variable in each startup script so I can tell if it has been sourced. .bashrc will set DOTBASHRC=1; .profile will set DOTPROFILE=1, etc.
With this set up, I ran
ssh macosx env
and got this:
DOTBASHRC=1
Ah, empiricism!
Problem solved. In order to get PATH set the way you want it for remote ssh command execution, you have to set it in your .bashrc on the remote machine.
Phew!
(Extra credit homework: What does csh do?)