How do you list number of lines of every file in a directory in human readable format.

Question

I have a list of directories and subdirectories that contain large csv files. There are about 500 million lines in these files, each is a record. I would like to know

1. How many lines are in each file.
2. How many lines are in directory.
3. How many lines in total

Most importantly, I need this in 'human readable format' eg. 12,345,678 rather than 12345678

It would be nice to learn how to do this in 3 ways. Plain vanilla bash tools, awk etc., and perl (or python).

Answer 1

How many lines are in each file.

Use wc, originally for word count, I believe, but it can do lines, words, characters, bytes, and the longest line length. The -l option tells it to count lines.

wc -l <filename>

This will output the number of lines in :

$ wc -l /dir/file.txt
32724 /dir/file.txt

You can also pipe data to wc as well:

$ cat /dir/file.txt | wc -l
32724
$ curl google.com --silent | wc -l
63

How many lines are in directory.

Try:

find . -name '*.pl' | xargs wc -l

another one-liner:

( find ./ -name '*.pl' -print0 | xargs -0 cat ) | wc -l

BTW, wc command counts new lines codes, not lines. When last line in the file does not end with new line code, this will not counted.

You may use grep -c ^ , full example:

#this example prints line count for all found files
total=0
find /path -type f -name "*.php" | while read FILE; do
     #you see use grep instead wc ! for properly counting
     count=$(grep -c ^ < "$FILE")
     echo "$FILE has $count lines"
     let total=total+count #in bash, you can convert this for another shell
done
echo TOTAL LINES COUNTED:  $total

How many lines in total

Not sure that I understood you request correctly. e.g. this will output results in the following format, showing the number of lines for each file:

# wc -l `find /path/to/directory/ -type f`
 103 /dir/a.php
 378 /dir/b/c.xml
 132 /dir/d/e.xml
 613 total

Alternatively, to output just the total number of new line characters without the file by file counts to following command can prove useful:

# find /path/to/directory/ -type f -exec wc -l {} \; | awk '{total += $1} END{print total}'
 613

Most importantly, I need this in 'human readable format' eg. 12,345,678 rather than 12345678

Bash has a printf function built in:

printf "%0.2f\n" $T

As always, there are many different methods that could be used to achieve the same results mentioned here.

Answer 2

In many cases combining the wc command and the wildcard * may be enough.
If all your files are in a single directory you can call:

wc -l src/*

You can also list several files and directories:

wc -l file.txt readme src/* include/*

This command will show a list of the files and their number of lines.
The last line will be the sum of the lines from all files.

---

To count all files in a directory recursively:

First, enable globstar by adding shopt -s globstar to your .bash_profile. Support for globstar requires Bash ≥ 4.x which can be installed with brew install bash if needed. You can check your version with bash --version.

Then run:

wc -l **/*

Note that this output will be incorrect if globstar is not enabled.

Answer 3

This command will give list of lines code in each directory:

find . -name '*.*' -type f | xargs wc -l

Answer 4

I will just augment @malyy answer for the following (to big for a comment):

How many lines in total

Many answers are using wc command line file option with xargs. The problem with this is xargs is limited to a rather small platform dependent size.

Furthermore there is a difference between BSD (macOS) and GNU (linux/homebrew) wc.

The GNU one is ideal because it can read the file listing from a file instead of arguments (--files0).

If you are on mac and have homebrew you should do the following:

find . -name "*.pl" -print0 | gwc -l --files0=-

Notice the gwc instead of wc.

Answer 5

a bit late to the game, but I got a bunch of argument errors with the above due to the size of the dir. This worked for me:

for i in $(find . -type f); do wc -l $i; done >> /home/counts.txt

Answer 6

cat would combine the files into one and outputs everything to stdout, you can do a wc -l on that for a total count of lines of files in a directory:

cat /path/to/directory/* | wc -l

Answer 7

If you need only total lines from all files use grep

find . -name '*.csv' | xargs wc -l | grep total

This command works too

wc -l *.csv | grep total

Answer 8

On the theme of “another way to do it, not necessarily the best way”, how about:

find /path/to/directory -name '*' -exec wc {} +

This is ok for white space in the filename but still suffers from the “too many files” problem.