When (or why) should you check for a file's existence before sourcing it?

Question

When attempting to source a file, wouldn't you want an error saying the file doesn't exist so you know what to fix?

For example, nvm recommends adding this to your profile/rc:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm

With above, if nvm.sh doesn't exist, you'll get a "silent error". But if you try . "$NVM_DIR/nvm.sh", the output will be FILE_PATH: No such file or directory.

You shouldn't. It's racy. Try source then handle the error, if any — Mikel, Jun 17 '18 at 01:21
It's always better to check for existence first, imo. Just trying to open, and then depending on somehow being able to handle the error is ugly and sloppy. Unfortunately, it's also quite common. For example, in Python circles. I believe in some cases people think they should do this for performance reasons, but if so, that's just deranged. — Faheem Mitha, Jun 17 '18 at 06:13
@FaheemMitha, well, if what you're doing is at all security-sensitive (i.e. your program works on someone else's behalf), then you really do need to care about the race conditions (TOCTOU). That's probably not the case here, though, since you have bigger problems if someone gets to modify files in HOME. — ilkkachu, Jun 17 '18 at 06:58
@FaheemMitha It is never better to check for existence first. The situation can change between test and use, yielding both false positives and false negatives: and you still have to handle the failure on use anyway. And as you have mentioned performance, testing before use is endemically twice as slow as not doing so and letting the system do it, which it is going to do anyway. It can't not be. — user207421, Jun 17 '18 at 09:54
@ilkkachu Agreed, but it's not just about security, it's about correctness and robustness. Random example: what if you're modifying your nvm install, and you also have a cron job that sources .bash_profile. — Mikel, Jun 17 '18 at 14:30
Quite often, this pattern also means that the file is optional. — Simon Richter, Jun 17 '18 at 18:24
@SimonRichter, in this case nvm won’t work. User would have to figure out that the file is missing on their own. — JBallin, Jun 17 '18 at 20:40
If you just source something and that code exists, but generates an error, then simple code will not know the difference between that and it not existing at all. — Joe, Jun 23 '18 at 05:39

Stéphane Chazelas · Accepted Answer · 2018-06-20T16:47:31.737

25

In POSIX shells, . is a special builtin, so its failure causes the shell to exit (in some shells like bash, it's only done when in POSIX mode).

What qualifies as an error depends on the shell. Not all of them exit upon a syntax error when parsing the file, but most would exit when the sourced file can't be found or opened. I don't know of any that would exit if the last command in the sourced file returned with a non-zero exit status (unless the errexit option is on of course).

Here doing:

[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"

Is a case where you want to source the file if it's there, and don't if it's not (or is empty here with -s).

That is, it should not be considered an error (fatal error in POSIX shells) if the file is not there, that file is considered an optional file.

It would still be a (fatal) error if the file was not readable or was a directory or (in some shells) if there was a syntax error while parsing it which would be real error conditions that should be reported.

Some would argue that there's a race condition. But the only thing it means would be that the shell would exit with an error if the file is removed in between the [ and ., but I'd argue it's valid to consider it an error that this fixed path file would suddenly vanish while the script is running.

On the other hand,

command . "$NVM_DIR/nvm.sh" 2> /dev/null

where command¹ removes the special attribute to the . command (so it doesn't exit the shell on error) would not work as:

it would hide .'s errors but also the errors of the commands run in the sourced file
it would also hide real error conditions like the file having the wrong permissions.

Other common syntaxes (see for instance grep -r /etc/default /etc/init* on Debian systems for the init scripts that haven't been converted to systemd yet (where EnvironmentFile=-/etc/default/service is used to specify an optional environment file instead)) include:

[ -e "$file" ] && . "$file"

Check the file it's there, still source it if it's empty. Still fatal error if it can't be opened (even though it's there, or was there). You may see more variants like [ -f "$file" ] (exists and is a regular file), [ -r "$file" ] (is readable), or combinations of those.
[ ! -e "$file" ] || . "$file"

A slightly better version. Makes it clearer that the file not existing is an OK case. That also means the $? will reflect the exit status of the last command run in $file (in the previous case, if you get 1, you don't know whether it's because $file didn't exist or if that command failed).
command . "$file"

Expect the file to be there, but don't exit if it can't be interpreted.
[ ! -e "$file" ] || command . "$file"

Combination of the above: it's OK if the file is not there, and for POSIX shells, failures to open (or parse) the file are reported but are not fatal (which may be more desirable for ~/.profile).

^{¹ Note: In zsh however, you can't use command like that unless in sh emulation; note that in the Korn shell, source is actually an alias for command ., a non-special variant of .}

edited Jun 20 '18 at 16:47

answered Jun 17 '18 at 06:15

Stéphane Chazelas

544,893

Interesting! I didn't know that about POSIX sh. But the question was about .bash_profile. I guess better safe than sorry, but is bash ever in POSIX mode when .bash_profile is sourced? – Mikel Jun 17 '18 at 14:25
(I realize you could interpret this question as applying more broadly to all POSIX shells based on reading the nvm source link in the question.) – Mikel Jun 17 '18 at 14:26
@Mikel, my answer still applies to bash when not in POSIX mode. You'd want [ -e /file ] && . /file if you don't consider it to be an error when the file doesn't exist. The try source then handle the error, if any can't be done here. – Stéphane Chazelas Jun 17 '18 at 14:43
POSIX mode has corner cases where users can't log in if the file is deleted racily or the sourced file has a syntax error. Since we can't prevent that, my choice would be to require bash/zsh/ksh. – Mikel Jun 17 '18 at 17:13
For bash mode, "handle the error" is certainly hard since the exit status of . would be 1, which could mean that . failed or that it succeeded but the last command in the sourced file exited 1. But the original code doesn't handle that either. So . ./file || true at least "handles" the error in the same way the original does, while also ensuring the user can still log in if errexit is in effect and something in the sourced file causes a non-zero exit – Mikel Jun 17 '18 at 17:16
In this case, I would just . "$NVM_DIR/nvm.sh" || echo "Error sourcing $NVM_DIR/nvm.sh". – Mikel Jun 17 '18 at 17:19
1

@Mikel, that's counter-productive. 1) that does not prevent the exit upon error with POSIX shells (or bash in POSIX mode) 2), that duplicates the error message (yours on stdout), . will already report an error (on stderr). And if the intent is to not consider it an error when the file doesn't exist, that's not correct (and it's not possible, from the exit status to tell if . failed because the file didn't exist or was not readable, or was not parsable, or the last command failed) which are the points I'm making here in this answer. – Stéphane Chazelas Jun 17 '18 at 18:08
1

With respect to the race condition -- it's much less of a problem IMHO for login to fail once (while something weird is happening on the system) than for login to fail consistently (even if there's something not-quite-right in the user's config-files). So a race condition in this check is still an improvement over not having the check. – ruakh Jun 18 '18 at 04:33
@StéphaneChazelas My point was to require bash in non-POSIX mode, then we can handle things 100% correctly. Since the question was about .bash_profile, that should be a reasonable requirement. – Mikel Jun 20 '18 at 16:24
@StéphaneChazelas Yes, I already acknowledged we can't tell the difference between file missing and the sourced file exiting 1 in my earlier comment (..."exit status of . would be 1"...). – Mikel Jun 20 '18 at 16:30
@ruakh My point was that we should never cause login to fail. This is why I said we should require non-POSIX bash, since nothing else needs bash POSIX mode in this case AFAICS. – Mikel Jun 20 '18 at 16:32
@Mikel, even in non-POSIX mode bash would still output an error message if the file didn't exist, and if you add a 2> /dev/null, then you hide the real errors. The OP mentions that he sees this kind of thing everywhere, and it's true it's found in many scripts like init scripts that do some [ -e /etc/default/foo ] && . /etc/default/foo, where often the /etc/default/foo is not there by default but can be created by the admin to add some customisation. – Stéphane Chazelas Jun 20 '18 at 16:38
@StéphaneChazelas I see the nvm author's intent was to silently do nothing if nvm was not installed. Personally, I would like to know if I'm trying to enable nvm but it's not available for some reason. If I'm not trying to enable nvm, I can remove that code from my .bash_profile or add a test that only enables nvm when needed. The error message you seem to say is redundant is mostly a question of style. I lean towards emitting an error message since the error could be anywhere in the sourced file too. But it depends on the intended use case, and reasonable people can disagree. :) – Mikel Jun 20 '18 at 17:27

JBallin · Answer 2 · 2018-06-17T17:26:10.700

5

Maintainer of nvm's response:

it's easy to uninstall nvm by simply deleting the file; forcing additional work (to track down where the line(s) are that source nvm) doesn't seem particularly valuable.

My interpretation (combined with Stéphane’s excellent explanation and Kusalananda’s comment):

It’s simpler and safer.

It defends against POSIX shells exiting on startup due to a missing file (for various reasons). Those using non-POSIX (e.g. bash) shells can remove the conditional if they prefer.

edited Jun 17 '18 at 17:26

answered Jun 17 '18 at 07:32

JBallin

371

1

I oppose your interpretation that it's "aimed at beginners". It's defensive. You don't want a user's login shell to terminate unexpectedly on startup just because that file has gone missing (for whatever reason). If that line of code is in one of the shell init files under /etc, then it allows for some users having the file, and some not having the file. IMHO, the nvm maintainer's response is only touching on one aspect. – Kusalananda Jun 17 '18 at 08:07
POSIX support is a good justification, but IMO "simpler and safer" is inaccurate. It's more just like "lazier" because any attentive sysadmin should A) want to know if there are any errors and B) seek to fix those errors even if they are minor... in the case of nvm it is a third-party extension and they really don't want to break anyone's system, so it's probably not a very good example here when discussing general best practices re: sourcing files. – Jesse Nickles Feb 21 '21 at 19:11

Mikel · Answer 3 · 2018-06-20T17:33:24.493

As JBallin and Stéphane Chazelas have pointed out, in POSIX shells, sourcing a file that doesn't exist would cause logging in to fail.

But adding a test to see if the file exists and then trying to source it can cause something called a race condition. If something changes nvm.sh in between the [ -s nvm.sh ] and the . nvm.sh, it will cause exactly the bug they're trying to prevent, albeit much more rarely.

In general, the way to prevent race conditions is to just try the thing you want to do, then handle the error if it fails, e.g.

. "$NVM_DIR/nvm.sh" || echo "Sourcing $NVM_DIR/nvm.sh failed" >&2

It turns out this doesn't work in POSIX shells, because, as above, . failing will cause the shell to exit immediately, before any error handling can run.

My answer argues POSIX shells are not relevant to this question, because .bash_profile shouldn't ever run in POSIX mode. So we can just do the code above anyway.

To be safest, we could ensure that POSIX mode is not in effect, or ensure POSIX mode is disabled using the technique described in https://unix.stackexchange.com/a/383581/3169.

Stéphane's answer has some useful suggestions for how to handle all POSIX shells, which I think was the nvm author's intent, but was subtly different than what the question here was asking, which is why we have multiple possible approaches, depending on what your goal is.

When (or why) should you check for a file's existence before sourcing it?

3 Answers3