Find files that are not in .gitignore

Question

I have find command that display files in my project:

find . -type f -not -path './node_modules*' -a -not -path '*.git*' \
       -a -not -path './coverage*' -a -not -path './bower_components*' \
       -a -not -name '*~'

How can I filter the files so it don't show the ones that are in .gitignore?

I thought that I use:

while read file; do
    grep $file .gitignore > /dev/null && echo $file;
done

but .gitignore file can have glob patterns (also it will not work with paths if file is in .gitignore), How can I filter files based on patterns that may have globs?

man find says: "-path pattern ... To ignore a whole directory tree, use -prune" so you want something like find . -name .git -prune -o -name node_modules -prune -o -type f (-o means "or") — milahu, Apr 02 '21 at 06:42
@MilaNautikus the question was about gitignore not about find, that was just example. — jcubic, Apr 02 '21 at 08:38

score 17 · Accepted Answer · answered Apr 11 '17 at 10:25

17

git provides git-check-ignore to check whether a file is excluded by .gitignore.

So you could use:

find . -type f -not -path './node_modules*' \
       -a -not -path '*.git*'               \
       -a -not -path './coverage*'          \
       -a -not -path './bower_components*'  \
       -a -not -name '*~'                   \
       -exec sh -c '
         for f do
           git check-ignore -q "$f" ||
           printf '%s\n' "$f"
         done
       ' find-sh {} +

Note that you would pay big cost for this because the check was performed for each file.

answered Apr 11 '17 at 10:25

cuonglm

153,898

What is find-sh {} + at the end? – jcubic Apr 11 '17 at 10:57
@jcubic see http://unix.stackexchange.com/q/12902/38906 – cuonglm Apr 11 '17 at 10:58
Ok, and what's find-sh? – jcubic Apr 11 '17 at 11:04
@jcubic it's the name you gave for inline-sh script, so in case error occurs, you would have an meaningful error message – cuonglm Apr 11 '17 at 11:07
@jcubic Try find /tmp -exec sh -c ': "${non_existed_var:?foo}"' find-sh {} + for example – cuonglm Apr 11 '17 at 11:12
This solution is good, but need git. It would be awesome to have a generic solution for ignore files (.gitignore, .svnignore...). – Soullivaneuh Jan 31 '20 at 20:28
Works as expected, but there is a typo. As the whole for loop is put into single quotes, we need to change printf '%s\n' "$f" to printf "%s\n" "$f", otherwise \n is output as n. – tukusejssirs Mar 20 '21 at 13:30
Didn't git grep used to just work and ignore the things listed in .gitignore? I feel like I remember it did. Now it doesn't. – mathtick Mar 12 '24 at 16:51

score 16 · Answer 2 · answered Apr 11 '17 at 10:35

16

To show files that are in your checkout and that are tracked by Git, use

$ git ls-files

This command has a number of options for showing, e.g. cached files, untracked files, modified files, ignored files etc. See git ls-files --help.

answered Apr 11 '17 at 10:35

Kusalananda

333,661

I also think about git ls-files but don't think it has option to show untracked files, which one? – cuonglm Apr 11 '17 at 10:41
2

@cuonglm -o (other). E.g., git ls-files -o -X .gitignore – Kusalananda Apr 11 '17 at 10:42
Ah yes, but can't handle the case where the file was tracked before but include in .gitignore. – cuonglm Apr 11 '17 at 10:47
Minor note: this does not list the actual files from the system but the files included in the git history. This can be a subtle difference on case-insensitive systems (e.g. Windows shares). Say, I committed ./foo.txt but then rename it to ./Foo.txt. Some git clients will not recognize this change and git ls-files outputs ./foo.txt while find would output ./Foo.txt – Gerrit-K Aug 29 '17 at 08:07

score 15 · Answer 3 · edited Sep 27 '17 at 12:16

15

there is a git command for doing exactly this: e.g.

my_git_repo % git grep --line-number TODO                                                                                         
desktop/includes/controllers/user_applications.sh:126:  # TODO try running this without sudo
desktop/includes/controllers/web_tools.sh:52:   TODO: detail the actual steps here:
desktop/includes/controllers/web_tools.sh:57:   TODO: check if, at this point, the menurc file exists. i.e. it  was created

As you stated, it will do a basic grep will most of the normal grep options, but it will not search .git or any of the files or folders in your .gitignore file.
For more details, see man git-grep

Submodules:

If you have other git repos inside this git repo, (they should be in submodules) then you can use the flag --recurse-submodules to search in the submodules as well

edited Sep 27 '17 at 12:16

KNejad

185

answered Apr 11 '17 at 10:25

the_velour_fog

12,270

Great, it almost work because I have other git repos inside current directory, that's why I have "*.git". – jcubic Apr 11 '17 at 10:53
So I'm seeing this has stopped working for some reason. v 1.40 – mathtick Mar 12 '24 at 16:52
What I mean is that ignored but tracked files are not ignored. – mathtick Mar 12 '24 at 16:53

score 6 · Answer 4 · answered Dec 27 '21 at 10:12

6

I think this works well:

git ls-files --cached --modified --other --exclude-standard

If you also want to recurse into submodules, add --recurse-submodules.

answered Dec 27 '21 at 10:12

Mitar

636

This works only for files that are tracked by git, the question doesn't say that it should only be files that are already in git. – jcubic Dec 27 '21 at 10:28
3

No, --other also outputs those files which are not tracked by git (and then --exclude-standard removes those ignored by git). – Mitar Dec 27 '21 at 10:31
It seems that files that are modified are duplicated on the list. So I would add | uniq at the end. – jcubic Dec 28 '21 at 11:34
1

Then maybe just --modified should not be listed there? – Mitar Dec 29 '21 at 00:13
--deduplicate should help with that. – Toby Speight Jul 10 '23 at 15:54
--recurse-submodules doesn't seem compatible with all those flags – AlexLoss Jul 11 '23 at 16:08

score 2 · Answer 5 · answered Apr 11 '17 at 10:20

2

You can use an array in which bash glob will be performed.

Having files like this :

touch file1 file2 file3 some more file here

And having an ignore file like this

cat <<EOF >ignore
file*
here
EOF

Using

arr=($(cat ignore));declare -p arr

Will result to this:

declare -a arr='([0]="file" [1]="file1" [2]="file2" [3]="file3" [4]="here")'

You can then use any technique to manipulate those data.

I personally prefer something like this:

awk 'NR==FNR{a[$1];next}(!($1 in a))'  <(printf '%s\n' "${arr[@]}") <(find . -type f -printf %f\\n)
#Output
some
more
ignore

answered Apr 11 '17 at 10:20

George Vasiliou

7,913

gitignore is superset of bash glob, you will miss for something like !file* – cuonglm Apr 11 '17 at 10:29
@cuonglm What do you mean by !file* ? As a glob pattern in ignore file or as a filename? – George Vasiliou Apr 11 '17 at 10:37
gitignore accept file pattern like !file* to exclude files start with file, and also double star ** – cuonglm Apr 11 '17 at 10:39
.gitignore has a bunch of features, all of which might not be that easy to implement directly in Bash: https://git-scm.com/docs/gitignore . Though ignores consisting of just simple globs should be doable in plain Bash, without awk or such – ilkkachu Apr 11 '17 at 10:42
@cuonglm You mean translation of ! as not ? Maybe extended globing can handle this situation. About doublestar , bash can handle it with shopt -s globstar. – George Vasiliou Apr 11 '17 at 10:43
No, you can't do it even with extended glob. it's !(pattern) in bash, not !pattern. And also for double star, like !**, /** – cuonglm Apr 11 '17 at 10:56
@cuonglm You are right about extglob. It has to be like !(pattern) and not !pattern. On the other hand, using !file in git maybe this means that git tools will fail to catch filenames starting with ! ? – George Vasiliou Apr 11 '17 at 11:05
@GeorgeVasiliou then escape ! to match ! literally. – cuonglm Apr 11 '17 at 11:08
@cuonglm OK. In git tools someone needs to escape ! , in bash tools we need to add ( ) . In both cases you need a kind of "processing". In any case, since we talk about .gitignore there is no doubt that git tools should and will perform better than bash. – George Vasiliou Apr 11 '17 at 11:10

score 0 · Answer 6 · edited Jul 10 '23 at 14:05

This solution uses metaprogramming heavily, but it's much faster than the answer marked correct by reducing repeated shell-command calls.

Replace sed deletion pattern with a substantial subset of your own known working-directory-level (./-starting) exclude list. Put the *-starting ones in the sed substitute pattern like normal find invocation (proper escaping plz).

FILES="$( \
  find . -depth 1 \
  | sed -e '/\/\.git/d' -e '/\/node_modules/d' \
      -e 's/.*/find '"'&'"' -type f/' \
  | sh \
)"
# >&2 echo debug: $FILES
HUGE_SED_COMMAND="$( 

  echo '/^('$( 

    git check-ignore --stdin <<<"$FILES" 

    | sed 's/./\./g; s///\//g' 

    )')$/d' 

  | sed 's/ /|/g' 

)"
>&2 echo debug: $HUGE_SED_COMMAND
sed -E "${HUGE_SED_COMMAND}" <<<"$FILES"

Find files that are not in .gitignore

6 Answers6

>&2 echo debug: $HUGE_SED_COMMAND