18

I was surprised recently when I did something like mv ./* ../somedirectory and found that files like .gitignore were not moved.

I do most of my work in zsh on OS X, and this surprise bit me in bash on CentOS. I tried bash on OS X and found the same behavior: * does not match dot files. This seems very undesirable to me, but apparently it's the bash default. (It may be the zsh default too for all I remember, but I may have changed it years ago in my .zshrc and forgotten it ever worked differently.)

How can I configure bash to behave as I expected: for * to match all files, and not ignore dot files.

In case this is at all unclear, here's how to reproduce it

cd /tmp
mkdir {t,d}est
touch test/{.,}{1,2,3,4,5,6,7}
ls -hal test
mv test/* dest
ls -hal test     # notice dot files are still there
ls -hal dest     # notice only some files were mv'ed
iconoclast
  • 9,198
  • 13
  • 57
  • 97
  • 1
    Yes, they're related. That didn't show up when I searched (probably because the questioner of that didn't mention globbing or dot files), but it's really a different question. He was asking how to move the files, and I was asking how to change the shell behavior, and specifically for bash. I might not have asked if that question had shown up for me (since your extremely complete and thorough answer there includes the bash setting) but it's still a different question, and someone looking for an answer to my question isn't necessarily going to find his question. – iconoclast Jun 18 '12 at 14:41
  • That whole “someone looking for an answer to my question isn't necessarily going to find his question” business is exactly why we have this way of closing questions as duplicates. – Gilles 'SO- stop being evil' Jun 18 '12 at 22:05
  • yes, but you seem to be missing the main point, which is that it's a different question. – iconoclast Oct 25 '16 at 20:14

5 Answers5

19

Bash

As you already noticed bash won't match a . at the start of the name or a slash. To change the matching regarding the dot you have to set the dotglob option - man bash:

dotglob If set, bash includes filenames beginning with a `.'  in
    the results of pathname expansion.

To enable/set it with bash use shopt, e.g:

shopt -s dotglob


For zsh you can also use the dotglob option but but you will have to use setopt to enable it, e.g:

setopt dotglob
Ulrich Dangel
  • 25,369
14

* is a glob that is expanded by the shell. By default shells don't include files whose name starts with a . (called hidden files or dotfiles) unless the leading . is entered literally.

* or [.]* or ?* or *.* or dir/* will not include dotfiles.

.* or dir/.* will.

So you could do:

mv -- * .* /dest/

however some shells including bash (but not zsh, mksh nor fish) have that misfeature that the expansion of .* include the . and .. special directory entries, which you don't want here (and generally never want a glob to include which is why I call it a misfeature).

For that reason, you'll find that sometimes people use (in Bourne-like shells):

mv -- * .[!.]* ..?* /dest/

That's three globs, the first one matching non-hidden files, the second one filenames starting with . followed by a character other than . and the 3rd one filenames starting with .. followed by at least one character.

However some modern shells have better ways around that

zsh

With zsh, you can use the (D) glob qualifier to specify that the glob should include dotfiles:

mv -- *(D) /dest/

zsh also fixed that other misfeature of the Bourne shell in that if the pattern doesn't match, the mv command is not run.

As said above it also will never include . nor .. in its globs, so

mv -- * .* /dest/

will be safe. However if there's no file matching * or no file matching .* the command will be aborted, so it would be better to use:

mv -- (*|.*) /dest/

Like in some other shells, you can also force all globs to include dotfiles (for instance if you find yourself wanting dotfiles included more often than not) with:

setopt dotglob

or:

set -o dotglob

After that, if you want a particular glob not to include dotfiles, you can write it:

echo *(^D)

Or:

echo [^.]*

Bash

Unfortunately bash doesn't have glob qualifiers. So you're left with enabling dotfile inclusion globally. In bash, the syntax is:

shopt -s dotglob

(and use [^.]* for globs without hidden files).

With dotglob, bash doesn't include . nor .. in globs like *, but still does for globs like .*.

If you set the GLOBIGNORE variable to something non-empty, it then automatically enables the dotglob option and excludes . and .. from .* globs but not from dir/.* or .*/file ones (!) so that safeguard is pretty useless. You could do GLOBIGNORE='*/.:*/..:./*:../*:*/./*:*/../*' but then it would break globs like */. or ./* or ../*.

A better work around is to use [.]* or dir/[.]* or [.]*/file (with dotglob enabled) to expand dotfiles except . and ...

fish

fish globs don't include . nor ... When there's no match, depending on the version, it will work like either zsh (or bash -o failglob) or bash -o nullglob.

mv -- * .* /dest/

Would work if there are both hidden and non-hidden files. Otherwise, YMMV and with some versions, it may call mv -- /dest if there's no file at all.

ksh93

No glob qualifier in ksh93 either. You can include dotfiles in globs with:

FIGNORE='@(.|..)'

Contrary to bash's GLOBIGNORE, that's done properly and also fixes the problem of .* including . and ...

yash

yash has a dot-glob option (set -o dot-glob), but contrary to bash, the glob expansions (even of *) include . and .. so it's pretty useless.

tcsh

set globdot

Works like in bash, that is * include dot files except . and .. but .* still includes . and .. (and you can use [.]* to expand hidden files except . and ..).

7

I tested this and it solves the issue:

shopt -s dotglob

Output:

~/stackexchangeanswers/40662$ ls -hal dest
total 8.0K
drwxr-xr-x 2 jodiec jodiec 4.0K 2012-06-12 22:15 .
drwxr-xr-x 4 jodiec jodiec 4.0K 2012-06-12 22:15 ..
-rw-r--r-- 1 jodiec jodiec    0 2012-06-12 22:15 1
-rw-r--r-- 1 jodiec jodiec    0 2012-06-12 22:15 .1
-rw-r--r-- 1 jodiec jodiec    0 2012-06-12 22:15 2
-rw-r--r-- 1 jodiec jodiec    0 2012-06-12 22:15 .2
-rw-r--r-- 1 jodiec jodiec    0 2012-06-12 22:15 3
-rw-r--r-- 1 jodiec jodiec    0 2012-06-12 22:15 .3
...snipped....
Jodie C
  • 1,879
4

The easiest way in bash to print matching dot files, ignoring . and .. is:

$ GLOBIGNORE=/+/
$ printf '%s\n' *


To test:

$ cd tmp; mkdir empty; cd empty; touch {,.}{a..h}
$ GLOBIGNORE=/+/
$  ls *
a  .a  b  .b  c  .c  d  .d  e  .e  f  .f  g  .g  h  .h

It printed all files with dots or not, with the notable exception of . and ...

Your example will also work correctly:

$ cd /tmp; mkdir {t,d}est; touch test/{,.}{a..h}
$ mv test/* dest/
$ ls -a test
.  ..
$ ls -a test/*
ls: cannot access test/*: No such file or directory

Empty of files that matter.
The system files . .. still exist but a shell expansion (using *) will not include them.

And the destination directory contains all of the files:

$ cd dest
$ ls -a *
a  .a  b  .b  c  .c  d  .d  e  .e  f  .f  g  .g  h  .h

Why?

This works because setting GLOBIGNORE has this side effects:

  • Set dotglob (shopt -s dotglob) to math dot files on expansions.
  • File names . and .. are ignored when GLOBIGNORE is set and not null.

Details in the manual: LESS=+'/The GLOBIGNORE' man bash.

Also, we need to set the variable GLOBIGNORE to contain a name that no filename could match:
an slash (and something else just as a double precaution).

$ GLOBIGNORE=/+/

It may be useful to also set nullglob if it is needed to avoid getting an * when there is no file to match the *.

0

Very interested this setting of GLOBIGNORE=/+/ @user79743

From my experience unsetting dotglob is bad business to almost all your commands (cp, mv etc.) BUT the same is true setting the nullglob (especially with regular expressions that then need escaping!).

Nevertheless, if you do need to set them in the beginning of your program but interfere in specific parts where you call commands like cp, mv etc. or use regular expressions just do this:

  # set dotglob, unset nullglob AFTER this
  is_nullglob=$( shopt -s | egrep -i '.*nullglob' )
  is_dotglob=$( shopt -s | egrep -i '.*dotglob' )
  [[ $is_nullglob ]] && shopt -u nullglob
  [[ ! $is_dotglob ]] && shopt -s dotglob
  # ... call commands ...
  # .....................
  # reset dotglob, nullglob to their previous values
  [[ $is_nullglob ]] && shopt -s nullglob
  [[ ! $is_dotglob ]] && shopt -u dotglob
centurian
  • 141
  • 1
  • 5