How Can I Expand A Tilde ~ As Part Of A Variable?

Question

When I open up a bash prompt and type:

$ set -o xtrace
$ x='~/someDirectory'
+ x='~/someDirectory'
$ echo $x
+ echo '~/someDirectory'
~/someDirectory

I was hoping that the 5th line above would have went + echo /home/myUsername/someDirectory. Is there a way to do this? In my original Bash script, the variable x is actually being populated from data from an input file, via a loop like this:

while IFS= read line
do
    params=($line)
    echo ${params[0]}
done <"./someInputFile.txt"

Still, I'm getting a similar result, with the echo '~/someDirectory' instead of echo /home/myUsername/someDirectory.

Answer 1

The POSIX standard imposes word expansion to be done in the following order (emphasize is mine):

1. Tilde expansion (see Tilde Expansion), parameter expansion (see Parameter Expansion), command substitution (see Command Substitution), and arithmetic expansion (see Arithmetic Expansion) shall be performed, beginning to end. See item 5 in Token Recognition.

2. Field splitting (see Field Splitting) shall be performed on the portions of the fields generated by step 1, unless IFS is null.

3. Pathname expansion (see Pathname Expansion) shall be performed, unless set -f is in effect.

4. Quote removal (see Quote Removal) shall always be performed last.

The only point which interests us here is the first one: as you can see tilde expansion is processed before parameter expansion:

1. The shell attempts a tilde expansion on echo $x, there is no tilde to be found, so it proceeds.
2. The shell attempts a parameter expansion on echo $x, $x is found and expanded and the command-line becomes echo ~/someDirectory.
3. Processing continues, tilde expansion having already been processed the ~ character remains as-is.

By using the quotes while assigning the $x, you were explicitly requesting to not expand the tilde and treat it like a normal character. A thing often missed is that in shell commands you don't have to quote the whole string, so you can make the expansion happen right during the variable assignment:

user@host:~$ set -o xtrace
user@host:~$ x=~/'someDirectory'
+ x=/home/user/someDirectory
user@host:~$ echo $x
+ echo /home/user/someDirectory
/home/user/someDirectory
user@host:~$

And you can also make the expansion occur on the echo command-line as long as it can happen before parameter expansion:

user@host:~$ x='someDirectory'
+ x=someDirectory
user@host:~$ echo ~/$x
+ echo /home/user/someDirectory
/home/user/someDirectory
user@host:~$

If for some reason you really need to affect the tilde to the $x variable without expansion, and be able to expand it at the echo command, you must proceed in two times to force two expansions of the $x variable to occur:

user@host:~$ x='~/someDirectory'
+ x='~/someDirectory'
user@host:~$ echo "$( eval echo $x )"
++ eval echo '~/someDirectory'
+++ echo /home/user/someDirectory
+ echo /home/user/someDirectory
/home/user/someDirectory
user@host:~$ 

However, be aware that depending on the context where you use such structure it may have unwanted side-effect. As a rule of thumb, prefer to avoid using anything requiring eval when you have another way.

If you want to specifically address the tilde issue as opposed to any other kind of expansion, such structure would be safer and portable:

user@host:~$ x='~/someDirectory'
+ x='~/someDirectory'
user@host:~$ case "$x" in "~/"*)
>     x="${HOME}/${x#"~/"}"
> esac
+ case "$x" in
+ x=/home/user/someDirectory
user@host:~$ echo $x
+ echo /home/user/someDirectory
/home/user/someDirectory
user@host:~$ 

This structure explicitly check the presence of a leading ~ and replaces it with the user home dir if it is found.

Following your comment, the x="${HOME}/${x#"~/"}" may indeed be surprising for someone not used in shell programming, but is in fact linked to the same POSIX rule I quoted above.

As imposed by the POSIX standard, quote removal happens last and parameter expansion happens very early. Thus, ${#"~"} is evaluated and expanded far before the evaluation of the outer quotes. In turns, as defined in Parameter expansion rules:

In each case that a value of word is needed (based on the state of parameter, as described below), word shall be subjected to tilde expansion, parameter expansion, command substitution, and arithmetic expansion.

Thus, the right side of the # operator must be properly quoted or escaped to avoid tilde expansion.

So, to state it differently, when the shell interpretor looks at x="${HOME}/${x#"~/"}", he sees:

1. ${HOME} and ${x#"~/"} must be expanded.
2. ${HOME} is expanded to the content of the $HOME variable.
3. ${x#"~/"} triggers a nested expansion: "~/" is parsed but, being quoted, is treated as a literal¹. You could have used single quotes here with the same result.
4. ${x#"~/"} expression itself is now expanded, resulting in the prefix ~/ being removed from the value of $x.
5. The result of the above is now concatenated: the expansion of ${HOME}, the literal /, the expansion ${x#"~/"}.
6. The end-result is enclosed in double-quotes, functionally preventing word splitting. I say functionally here because these double quotes are not technically required (see here and there for instance), but as a personal style as soon as an assignments gets anything beyond a=$b I usually find it clearer add double-quotes.

By-the-way, if look more closely to the case syntax, you will see the "~/"* construction which relies on the same concept as x=~/'someDirectory' I explained above (here again, double and simple quotes could be used interchangeably).

Don't worry if these things may seem obscure at the first sight (maybe even at the second or later sights!). In my opinion, parameter expansion are, with subshells, one of the most complex concept to grasp when programming in shell language.

I know that some people may vigorously disagree, but if you would-like to learn shell programming more in depth I encourage you to read the Advanced Bash Scripting Guide: it teaches Bash-scripting, so with a lot of extensions and bells-and-whistles compared to POSIX shell scripting, but I found it well written with loads of practical examples. Once you manage this, it is easy to restrict yourself to POSIX features when you need to, I personally think that entering directly in the POSIX realm is an unnecessary steep learning curve for beginners (compare my POSIX tilde replacement with @m0dular's regex-like Bash equivalent to get an idea of what I mean ;) !).

---

¹: Which leads me into finding a bug in Dash which don't implement tilde expansion here correctly (verifiable using x='~/foo'; echo "${x#~/}"). Parameter expansion is a complex field both for the user and the shell developers themselves!

Answer 2

One possible answer:

eval echo "$x"

Since you're reading input from a file, I would not do this.

You could search and replace the ~ with the value of $HOME, like this:

x='~/.config'
x=${x/#\~/${HOME}}
echo "$x"

Gives me:

/home/adrian/.config

EDIT: changed the search and replace to ${x/#\~/${HOME}} in order to only replace the initial ~. Thanks @user137369 for the suggestion.