Show sum of file sizes in directory listing

Question

The Windows dir directory listing command has a line at the end showing the total amount of space taken up by the files listed. For example, dir *.exe shows all the .exe files in the current directory, their sizes, and the sum total of their sizes. I'd love to have similar functionality with my dir alias in bash, but I'm not sure exactly how to go about it.

Currently, I have alias dir='ls -FaGl' in my .bash_profile, showing

drwxr-x---+  24 mattdmo  4096 Mar 14 16:35 ./
drwxr-x--x. 256 root    12288 Apr  8 21:29 ../
-rw-------    1 mattdmo 13795 Apr  4 17:52 .bash_history
-rw-r--r--    1 mattdmo    18 May 10  2012 .bash_logout
-rw-r--r--    1 mattdmo   395 Dec  9 17:33 .bash_profile
-rw-r--r--    1 mattdmo   176 May 10  2012 .bash_profile~
-rw-r--r--    1 mattdmo   411 Dec  9 17:33 .bashrc
-rw-r--r--    1 mattdmo   124 May 10  2012 .bashrc~
drwx------    2 mattdmo  4096 Mar 24 20:03 bin/
drwxrwxr-x    2 mattdmo  4096 Mar 11 16:29 download/

for example. Taking the answers from this question:

dir | awk '{ total += $4 }; END { print total }'

which gives me the total, but doesn't print the directory listing itself. Is there a way to alter this into a one-liner or shell script so I can pass any ls arguments I want to dir and get a full listing plus sum total? For example, I'd like to run dir -R *.jpg *.tif to get the listing and total size of those file types in all subdirectories. Ideally, it would be great if I could get the size of each subdirectory, but this isn't essential.

Answer 1

There's already a UNIX command for this: du

Just do:

du -bch 

As per convention you can add one or more file or directory paths at the end of the command. -h is an extension to convert the size into a human-friendly format, -b gives you the file size instead of disk usage, and -c gives a total at the end.

Answer 2

You can use du -h -c directory|tail -1

This will generate a single line with memory usage.

Answer 3

The following function does most of what you're asking for:

dir () { ls -FaGl "${@}" | awk '{ total += $4; print }; END { print total }'; }

... but it won't give you what you're asking for from dir -R *.jpg *.tif, because that's not how ls -R works. You might want to play around with the find utility for that.

Answer 4

with perl:

perl -le 'map { $sum += -s } @ARGV; print $sum' -- *.pdf

Size of all non-hidden PDF files in current directory.

Answer 5

Simply print the current line that you are summing the total of:

dir | awk '{ print; total += $4 }; END { print "total size: ",total }'

Answer 6

For counting files in a directory using a mask, I tend to follow this approach:

For bytes

du -ac --bytes  | grep "zip$" | awk '{ print; total += $1 }; END { print "total lobsters: ", total, " Bytes" }'

For Kibibytes

du -ac --bytes  | grep "zip$" | awk '{ print; total += $1 }; END { print "total lobsters: ", total/1024, " KiB" }'

For Mebibytes

du -ac --bytes  | grep "zip$" | awk '{ print; total += $1 }; END { print "total lobsters: " total/1024/1024 " MiB" }'

You get the idea.

The breakdown is simple:

• du - disk usage

• -a - all files
• -c - total bytes
• --bytes - print output in bytes [in newer versions of bash, not sure if this applies anymore]

• grep - global regular expression print [prints output matching patterns]

• "zip$" - pattern to match. 'zip' is the string, and the '$' denotes end of string/line/etc - in this case, match lines that END with 'zip'. Conversely, placing '^' at the start of the string indicates that the pattern will be at the start of the string [ie: "^start" will match lines beginning with the word 'start'] - With this knowledge, wrapping a string in ^ and $ respectively, will match lines that start/end with the pattern used. "^hello people$" will match strings saying 'hello people'. "^hello(.*)people$" will match strings saying 'hello french people' and 'hello coding people', but not 'hello coding people with no lives'

• awk - a scripting language programmed by Aho, Weinberger, and Kernighan. Not a very original name, but a very powerful language that is excellent for text processing and data extraction.

• { print; total += $1 }
  • print - print the line currently being iterated over
  • total += $1 - initialize variable total if it isn't already, and add the first block separated by the field separator, in this case a space character. This can be changed by the -F flag.
  • ; - line / statement terminator. you can put multiple awk statements on a single line using this, similar to terminating linux command line statements. Otherwise you could have them in a multi line thing still surrounded by { ... }
  • END - this effectively means that awk will execute the actions specified before it exits.
• { print "total lobsters: " total " Bytes" }
  • print "total lobsters: " - first part of string being output
  • total - the variable containing the total sum of lines iterated over
  • " Bytes" - final part of printed string, tacked on to the end of the prior two statements
  • obviously, these three statements are encapsulated in { } like the first part.

So, stepping through an example in a case where we want to count the total number of zip files in a directory:

du -ac --bytes

836544  ./wp-content/themes/astra.1.8.1.zip
934364  ./wp-content/themes/astra.2.0.1.zip
400033  ./wp-content/uploads/2019/09/premium-addons-for-elementor-3.2.9-WJdFQT1mLd3GA81lQEAo.zip
117351218       ./wp-content/uploads/backwpup-fc5928-temp/2019-05-30_00-47-01_TX6FSKC601.zip
1192275 ./wp-content/plugins/essential-addons-elementor-master.zip
170     ./wp-content/plugins/gravityforms/images/doctypes/icon_zip.gif
1969651 ./wp-content/plugins/acf.zip
4284    ./wp-content/plugins/types/application/controllers/api/handler/import_from_zip_file.php

Two components of output, col 1 represented by the numeric values: 836544, 934364... etc, and col 2 being the path of the file.

However, since there are two lines that do not match what we want - icon_zip.gif and import_from_zip_file.php - we want to exclude these. Since du does not provide a way to filter recursively by extension (that I know of), we filter using grep

grep "zip$"

This effectively has the output from du piped to it, and filters the lines that end in zip, eliminating the two records we don't want:

836544  ./wp-content/themes/astra.1.8.1.zip
934364  ./wp-content/themes/astra.2.0.1.zip
400033  ./wp-content/uploads/2019/09/premium-addons-for-elementor-3.2.9-WJdFQT1mLd3GA81lQEAo.zip
117351218       ./wp-content/uploads/backwpup-fc5928-temp/2019-05-30_00-47-01_TX6FSKC601.zip
1192275 ./wp-content/plugins/essential-addons-elementor-master.zip
1969651 ./wp-content/plugins/acf.zip

Then awk parses each line, with the numerics in col 1 being stored in $1

We get this:

836544  ./wp-content/themes/astra.1.8.1.zip
934364  ./wp-content/themes/astra.2.0.1.zip
400033  ./wp-content/uploads/2019/09/premium-addons-for-elementor-3.2.9-WJdFQT1mLd3GA81lQEAo.zip
117351218       ./wp-content/uploads/backwpup-fc5928-temp/2019-05-30_00-47-01_TX6FSKC601.zip
1192275 ./wp-content/plugins/essential-addons-elementor-master.zip
1969651 ./wp-content/plugins/acf.zip
total lobsters:  128.339  MiB

Answer 7

Adding the following to .bash_profile or .bashrc works for me.

dir ()
{
find . -iname "$@" -exec ls -lh {} \;
find . -iname "$@" -print0|xargs -r0 du -csh|tail -n 1;
}

Now when i do a dir *.mp3 it does recursively and prints total at the end. You can control how much depth you want by adding a maxdepth parameter to the find. I know running find twice is not a very effiecnt idea. But i couldnt think of a better way. Atleast it gets the job done.

Answer 8

Using du and a awk statement like the one mentioned above will provide what you are looking for.

Example: du /home/abc/Downloads/*.jpg | awk '{ print; total += $1 }; END { print "total size: ",total }'

This will list all files in folder Downloads of user abc ending in .jpg and prints the sum of all these files at the end of the listing.

Answer 9

To get both, dir output and size calculation, without using any of the other proposed options, you can use tee(1) and process substitution...

dir | tee >( awk '{ total += $4 }; END { print total }' )

Answer 10

tee will solve the problem of output disappearing on screen when it is piped to another command like awk.

So the command:

ls -FaGl | printf "%'d\n" $(awk '{SUM+=$4}END{print SUM}')

which only prints:

63,519,676,015

Is replaced with the command:

ls -FaGl | tee /dev/stderr | printf "%'d\n" $(awk '{SUM+=$4}END{print SUM}')

and now the full file listing appears with total:

total 62031069
drwxrwxrwx 1 rick      20480 Oct  9 15:47 ./
drwxrwxrwx 1 rick      12288 Jul 20  2020 ../
drwxrwxrwx 1 rick          0 Oct 15  2017 Captures/
-rwxrwxrwx 1 rick        504 Jun 29  2020 desktop.ini*
drwxrwxrwx 1 rick       4096 Nov 18  2017 Mass Effect Andromeda/
-rwxrwxrwx 1 rick  210355992 Nov  8  2019 Screencapture 2019-11-08 at 13.07.14.mp4*
-rwxrwxrwx 1 rick  127445089 Nov  8  2019 Screencapture 2019-11-08 at 13.43.55.mp4*
-rwxrwxrwx 1 rick  997439911 Nov 11  2019 simplescreenrecorder-2019-11-11_21.42.51.mkv*
   ( Long listing snipped... )
-rwxrwxrwx 1 rick 1546689758 Sep  6 22:35 simplescreenrecorder-2021-09-06_21.18.29*
-rwxrwxrwx 1 rick  422607080 Sep 18 19:13 simplescreenrecorder-2021-09-18_18.57.00*
63,519,676,015

TL;DR

Insert | tee /dev/stderr into your pipeline.

Total in human readable format

In my own ~/.bashrc is this function:

$ grep 'BytesToHuman(' -A20 ~/.bashrc
function BytesToHuman() {
# https://unix.stackexchange.com/questions/44040/a-standard-tool-to-convert-a-byte-count-into-human-kib-mib-etc-like-du-ls1/259254#259254

read StdIn
if ! [[ $StdIn =~ ^-?[0-9]+$ ]] ; then
    echo "$StdIn"       # Simply pass back what was passed to us
    exit 1              # Floats or strings not allowed. Only integers.
fi

b=${StdIn:-0}; d=''; s=0; S=(Bytes {K,M,G,T,E,P,Y,Z}iB)
while ((b > 1024)); do
    d="$(printf ".%02d" $((b % 1024 * 100 / 1024)))"
    b=$((b / 1024))
    let s++
done

echo "$b$d ${S[$s]}"
exit 0                  # Success!


} # BytesToHuman ()

So simply add | BytesToHuman to the end of the pipeline. Also remove printf builtin that was used previously:

ls -FaGl | tee /dev/stderr | awk '{SUM+=$4}END{print SUM}' | BytesToHuman

This will now display the total as:

59.15 GiB

If you would prefer to see 63.51 GB then the BytesToHuman() function needs to be changed from:

b=${StdIn:-0}; d=''; s=0; S=(Bytes {K,M,G,T,E,P,Y,Z}iB)
while ((b > 1024)); do
    d="$(printf ".%02d" $((b % 1024 * 100 / 1024)))"
    b=$((b / 1024))

To:

b=${StdIn:-0}; d=''; s=0; S=(Bytes {K,M,G,T,E,P,Y,Z}B)
while ((b > 1000)); do
    d="$(printf ".%02d" $((b % 1000 * 100 / 1000)))"
    b=$((b / 1000))

Answer 11

du path_to_your_files/*.jpg | awk '{ total += $1 }; END { print total }'

Answer 12

Just for the sake of clarity and completeness, here is my current iteration of the dir() function, as well as answers to some common comments (including the most highly-voted answer here).

function dir() {
    ls -FAGl --color=always "${@}" | awk '{
                print;
                total += $4
            }; END {
                if (total < 1024)
                    print "\t\ttotal: ",total;
                else if (total < (1024 * 1024))
                    print "\t\ttotal: ",total/1024,"KB";
                else if (total < (1024 * 1024 * 1024))
                    print "\t\ttotal: ",total/(1024*1024),"MB";
                else if (total < (1024 * 1024 * 1024 * 1024))
                    print "\t\ttotal: ",total/(1024*1024*1024),"GB";
            }'
}

Here's how it works:

-F appends an indicator (one of */=>@|) to entries

-A does not list the implied . and .. entities

-G omits group names (in systems that include them by default)

-l prints a long listing, which is what we need in order to get file sizes

--color=always should be self-explanatory

"${@}" passes any other arguments to ls

The awk script, which I cobbled together from several sources, including the accepted answer here, first prints a newline, then adds up all the numbers in column 4. That number is then tested to see if it's smaller than 1KB, 1MB, 1GB, and 1TB, respectively. It then prints the total with appropriate unit, indented by 2 tabs (16 spaces) which approximately aligns with the size column on most systems. This can be customized to your own needs.

---

Why do you call it dir? That's a GNU coreutil!

I named the function dir because I grew up using DOS, and when I started playing around with Linux (kernel 1.2!) a friend suggested I alias dir as ls -al as I had gotten so used to typing it. It's just muscle memory now. Eventually I found out about /usr/bin/dir, but I much prefer having long directory listings, so I never bothered changing my alias, and now function. If you use the dir program and/or don't want to shadow it, feel free to name the function whatever you want.

Why don't you use du?

I do, when all I'm interested in is how much space a directory takes up. I wanted to have this functionality when I'm doing directory listings as well (which I do much more frequently), so this function was born.

Answer 13

du * | awk -v sum=0 '{print sum+=$1}' | tail -1