Parallel and ls with white spaces

Question

This command works fine for dir with "regular" names as dir1 mydir my-dir, etc

ls | parallel 'echo -n {}" "; ls {}|wc -l'

give me the number of files for each dir

but for dir with white spaces like "my dir" or my long dir name don't work and give error.

How to quote/escape the white spaces?

Answer 1

GNU Parallel has no problems dealing with spaces:

$ mkdir 'a  b'
$ touch 'a  b/c  d'
$ ls | parallel 'echo -n {}" "; ls {}|wc -l'
a  b 1

It requires -0, if the names contain \n:

$ mkdir 'a
b'
$ touch 'a
b/c
d'
fails
$ ls | parallel 'echo -n {}" "; ls {}|wc -l'
$ parallel 'echo -n {}" "; ls {}|wc -l' ::: *
works
$ printf "%s\0" * | parallel -0 'echo -n {}" "; ls {}|wc -l'
$ parallel -0 'echo -n {}" "; ls {}|wc -l' ::: *

So what you see is probably due to your ls doing something weird: It may be an alias to ls --some-weird-option. Try \ls instead (or use printf ... | ... -0 or -0 ... ::: * as show above):

\ls | parallel 'echo -n {}" "; ls {}|wc -l'

(PS: Are you aware of --tag:

parallel --tag 'ls {}|wc -l' ::: *

)

Answer 2

Without the complications of parallel, xargs, or passing pathnames back and forth:

shopt -s nullglob dotglob
for dir in /; do
    set -- "$dir"/
    printf '%s:\t%s\n' "${dir%/}" "$#"
done

That is, iterate over all directories and count the number of names that the * glob expands to in each.

The above is for bash, and we set the nullglob shell option to ensure that non-matching patterns are removed instead of retained unexpanded. I'm also setting the dotglob shell option to be able to match hidden names.

The zsh shell can do it while at the same time filtering the glob matches for only directories (for the loop) and only, e.g., regular files (for the loop body). In the code below, the glob qualifier (ND/) makes the preceding * only match directories, with the same effect as with nullglob and dotglob set in the bash shell, and (ND.) makes the preceding * only match regular files in the same way.

for dir in *(ND/); do
    set -- $dir/*(ND.)
    printf '%s:\t%s\n' $dir $#
done

Would you want to do this recursively, to get the count of names in each directory in a hierarchy, then you could just plug in the above in find:

find . -type d -exec bash -O nullglob -O dotglob -c '
    for dir do
        set -- "$dir"/*
        printf "%s:\t%s\n" "$dir" "$#"
    done' bash {} +

(the above is a bit different from the plain bash loop at the start of this answer as this would never count names in directories accessed via symbolic links), or,

find . -type d -exec zsh -c '
    for dir do
        set -- $dir/*(ND.)
        printf "%s:\t%s\n" $dir $#
    done' zsh {} +

Answer 3

Assuming that you only want to list the number of regular files in the sub-directories from where you execute your cmd, here is a one-liner that will do that for you:

 $ find . -maxdepth 1 -type d ! -name "." -print0 2>/dev/null \
   | xargs -0 -I {} sh -c 'printf "%20s:  %d\n" "{}" "$(find "{}" -maxdepth 1 -type f 2>/dev/null| wc -l)"'

Example output:

              ./Maildir:  0
             ./.dvisvgm:  0
               ./.pyenv:  5
             ./.ipython:  0
   ./.ipynb_checkpoints:  3
                ./.tmux:  1
         ./.virtualenvs:  12
         ./seaborn-data:  2
               ./.local:  2
                ./bgpix:  12
                 ./.vim:  7
     ...

• I added 2>/dev/null in each find block only to avoid some unwanted file access issues on the platform I used to run tests. You may do away with it if you foresee no such file permission issues when stat'ing your files as part of your find cmds.
• I also suppress all output concerning $PWD (your present working directory, denoted .) in keeping with my assumption stated above, which was that you are only interested in counting regular files in current 1st-level subdirectories.
• to count regular files in your whole sub-directory tree, starting at $PWD, just omit the -maxdepth 1 global option in the first find block above (but keep it in the 2nd one).

To better highlight the similarity with the solution relying on parallel (below), the above can be rewritten:

$ xargs -0 -I {} sh -c 'printf "%20s:  %d\n" "{}" "$(find "{}" -maxdepth 1 -type f 2>/dev/null| wc -l)"' \
  < <(find . -maxdepth 1 -type d ! -name "." -print0 2>/dev/null)

Relying on parallel instead of xargs, as in the above, requires escaping some quotes as follows (output is exactly as before):

$ parallel -0 -I {} \
  'sh -c "printf \"%20s: %d\n\" \"{}\" \"$(find {} -maxdepth 1 -type f 2>/dev/null | wc -l)\""' \
  :::: < <(find -maxdepth 1 -type d ! -name "." -print0 2>/dev/null)

• xargs and parallel use the same two arguments -0 -I {} ,
• the shell command to execute is the Bourne shell between single quotes,
  'sh -c "printf ..."', and
• the input to parallelize, introduced by ::::, is the process substitution input file <(...) which contains the output of the "find all subdirectories at first sublevel starting at $PWD" cmd.

Answer 4

A workaround found using find instead of ls.

find * -type d -maxdepth 0 | parallel 'echo -n {}" "; ls {}|wc -l'