0

I have a file, which contains the name of some files. I want to find the total size of files in the list.

#cat filelist
/tmp-directory/connector_db_ connector_db
/tmp-directory/connector_db -connector_db
/tmp-directory/connector_db_connector_db

As you can see the names of the file contains space, and more badly a hyphen (-). I am trying to find the total size of these files with below command.

#du -sch `cat filelist`
du: invalid option -- 'o'
du: invalid option -- 'n'
du: invalid option -- 'n'
du: invalid option -- 'e'
du: invalid option -- 't'
du: invalid option -- 'o'
du: invalid option -- 'r'
du: invalid option -- '_'
du: invalid maximum depth `b'

I tried adding " " as shown below and then tried which also failed

#cat filelist
"/tmp-directory/connector_db_ connector_db"
"/tmp-directory/connector_db -connector_db"
"/tmp-directory/connector_db_connector_db"

#du -sch `cat filelist`
du: invalid option -- 'o'
du: invalid option -- 'n'
.....

But this works when I use below command, directly in shell.

#du -sch "/tmp-directory/connector_db -connector_db"
0       /tmp-directory/connector_db -connector_db
0       total

So, how to handle such a situation. I have filelist of 3 lac files. More badly, "du -sch" is not handling the list when file list exceeds around 20000. I am splitting the list into 20000 lines with split command. Is there any alternate method for finding the size of 3 lac files easily?

Gilles 'SO- stop being evil'
  • 829,060
Midhun Jose
  • 33

2 Answers2

1
xargs -Ifile du -sch file <filelist

As long as no filename contains a newline, the above will call du -sch on each of the name in filelist. -Ifile will cause xargs to replace the string file in the du -sch file command with each name read from filelist.

This will handle filenames with spaces and with shell globbing patterns properly (patterns will not be expanded).

Kusalananda
  • 333,661
  • Awsome..! Could you please suggest a way to find the total size of the output. A typical scenario will be below. 4.0K /mnt/shared/5 56K /mnt/shared/6 1.1M /mnt/shared/7 – Midhun Jose Jun 09 '17 at 05:48
  • @midhunpmj You can use -k (only) with du and get count in kilobytes, and then get the sum of column 1. Something like xargs ... | awk '{ print; s += $1 } END { print s }'. The output would be in kilobytes, make necessary conversions. – Kusalananda Jun 09 '17 at 06:03
0

As you have not put quotes around command substitution, word splitting is taking place here, according to the value of IFS (space, tab, newline by default). You have space in filename, so e.g. /tmp-directory/connector_db_ connector_db is being split into two words -- /tmp-directory/connector_db_, connector_db.

You can iterate over the filenames one by one:

while IFS= read -r file; do du -sch -- "$file"; done <filelist

As your file is big, this might take some time.

Also as the files are being parsed individually, you won't have the total size shown by du. So you can drop -s and -c:

while IFS= read -r file; do du -h -- "$file"; done <filelist
heemayl
  • 56,300
  • Awsome..! Could you please suggest a way to find the total size of the output. A typical scenario will be below.

    4.0K /mnt/shared/5 56K /mnt/shared/6 1.1M /mnt/shared/7

     – Midhun Jose Jun 09 '17 at 05:47
  • @midhunpmj You can get rid of -h and get count in bytes, and then get the sum of column 1. Something like while ... | tee >(awk '{s+=$1} END{print s}') The output would be in bytes, make necessary conversions. – heemayl Jun 09 '17 at 05:51