Getting a list of files by a pattern in a directory and all nested ones, and iterating over a result

Question

In a directory, "my_directory", which has other directories in it, of multiple level of nesting, there're some files that I want to find by pattern and then, one by one, upload them to a remote server. On a remote server they should be copied over to the same path relative to the remote my_directory. For that, I'll scp

I've started off with this:

ls -al my_directory | grep "*.my_custom."

and it's returned nothing, although the files with *.my_custom. part in them exist.

How can I:

a) find those files, by a pattern

b) enumerate them in a loop, in bash, so I can refer to each as a variable? In order to upload it to a server as a next step.

Please provide the file names that should be passed, add more details about your directory and relative path. What does path relative to remote my_directory mean? — Amit Singh, Oct 05 '20 at 07:33
See https://unix.stackexchange.com/questions/128985/why-not-parse-ls-and-what-to-do-instead — Panki, Oct 05 '20 at 08:36

AdminBee · Answer 1 · 2020-10-14T15:39:24.337

The problem with your approach is two-fold:

You are using globbing characters (a.k.a. "wildcards") in a grep call, although grep expects a regular expression. These two are distinct. In globs, * means "a string of any characters and any length", whereas in regular expressions it means "zero or more of the preceding character", or a literal * if there is no preceding character. Conversely, the . in a glob means a literal ., whereas in a regular expression it means "any single character". So, your grep would look for files that literally contain the asterisk, then any single character, then my_custom and again any single character. For the match you want to achieve, the regular expression would be
```
.*\.my_custom\.
```
or, since you didn't anchor it, simply
```
\.my_custom\.
```
You want to use the generated list in a shell script for further processing. As @Panki already noticed, you should never parse the output of ls for that purpose, as it will stumble on filenames with special characters. Fortunately, you can go back to your original attempt with globs and iterate over the files directly in the shell:
```
for file in *.my_custom.*
do
  your_upload_command "$file"
done
```
Notice the shell variable $file is placed in double quotes. This is to ensure that no unwanted word-splitting occurs, i.e. your shell script doesn't stumble on whitespace or any other special character in the filename.
Since you stated that you want to do this recursively, this can be achieved with a minor change to the shell script, by use of the globstar shell option, available since Bash v4:
```
shopt -s globstar
for file in **/*.my_custom.*
do
  your_upload_command "$file"
done
```

score 0 · Answer 2 · answered Oct 05 '20 at 09:48

This can be done via find.

find my_directory -name "*.my_custom." -type f -exec scp -i {} <username>@<host>:<path>/{}  \;

{} will contain the path of the matching file starting from my_directory. exec specifies the command to be executed for each match. You can modify your scp path, keeping these two points in mind.

score 0 · Answer 3 · answered Oct 05 '20 at 10:18

0

I'd do (on GNU systems):

(cd my_directory && find . -name '*.my_custom.*' -type f -print0) |
  tar --null -T - -cf - | gzip -1 | ssh host 'cd my_directory && tar zxf -'

answered Oct 05 '20 at 10:18

Stéphane Chazelas

544,893

Getting a list of files by a pattern in a directory and all nested ones, and iterating over a result

3 Answers3