run commands inside awk before print

Question

I wish to list directories while calculating their size and print the whole thing in a simple command, using ls and awk, work great to control the list and what I wish to display :

ls -AlhF /usr | awk '{print $6, $7, $8, "\011", $9 }'

But I would like to go a bit further and to get the size of the folders or the files, use $9 with the command du -s with something like this :

ls -AlhF /usr | awk '{print $6, $7, $8, "\011", $9, du -s /usr/$9 }'

I've tried many ways to write it, with backticks, doublequote, but I always end up with errors or unwanted results.

Answer 1

To answer the question and ignoring the issues about parsing the output of ls for now, to get the output (stdout) of a command in awk, you do:

cmd="that command"
output=""
while ((cmd | getline line) > 0)
  output = output line RS
status = close(cmd)

How the exit status is encoded in the status variable varies from one awk implementation to the next. All you can rely on is that it will be 0 if and only if cmd succeeds.

getline starts a shell (usually a POSIX shell, but on some systems, that could be a Bourne shell) to parse that command line. So it's important to properly quote data in there. Best is to use single quotes which is the safest.

Here since the output will be a single line anyway (your approach can't handle file names with any spacing characters, let alone newlines), you only need to do one getline:

 awk -v q="'" '
   function shquote(s) {
     gsub(q, q "\\" q q, s)
     return q s q
   }
   {
     cmd = "du -sk " shquote("/usr/" $9)
     cmd | getline du_output
     status = close(cmd)
   }'

If you call getline without a variable name, then it sets $0 which can make it easier to extract the disk usage:

DIR=/usr export DIR
LC_ALL=C ls -l -- "$DIR" | awk -v q="'" '
  function shquote(s) {
    gsub(q, q "\\" q q, s)
    return q s q
  }
  {
    date = $6 " " $7 " " $8
    name = $9
    cmd = "du -sk " shquote(ENVIRON["DIR"] (ENVIRON["DIR"] ~ /^\/*$/ ? "" : "/") name)
    cmd | getline
    disk_usage = $1
    print date "\t" name "\t" disk_usage
  }'

Answer 2

ls -AlhF /usr | awk '{print "echo "$6" "$7" "$8" \011 "$9" $(du -s /usr/"$9")" }' | sh

Construct an echo and du command with awk and then execute it through piping to sh.

Answer 3

Here's a perl implementation which doesn't rely on parsing the output of ls -l (which is inherently unreliable), and can easily be modified to use any date or size output format required.

Save it to a file, make it executable and run it as /path/to/script /usr. The script requires one or more directory name arguments.

#!/usr/bin/perl

use strict;
use Date::Format;

if ($#ARGV < 0) { die "Missing directory argument" };

foreach my $dir (@ARGV) {
    $dir =~ s:/+$::;  # strip trailing /
    opendir(my $dh, $dir) || die "Can't opendir $dir: $!";

    while (readdir $dh) {
      next if (m/^\.{1,2}$/); # skip . and .. entries

      my $mtime=(stat("$dir/$_"))[9];   # extract mtime field from stat
      my $mtime_f = time2str('%Y-%m-%d %X', $mtime);

      open(my $fh, "-|", 'du', '-s', '--', "$dir/$_");
      my $du = <$fh>;
      ($du) = split(' ',$du); # only want the first blank-separated field
      close($fh).

      printf "%s\t%s\t%s\n", $mtime_f, "$dir/$_" , $du;
    }

    closedir $dh;
};

There are simpler, shorter ways of doing this (in perl or awk or other languages) but this provides a useful base for reporting other information about directories and their contents. see perldoc -f stat for full details on the information available from the stat() function.

BTW, this could be implemented in awk, but you'd have to implement your own stat() function and a date-formatting function. Easier to use perl.

GNU awk aka gawk supports loadable modules and the filefuncs module provides a stat() which populates a statdata[] array. Use by adding @load "filefuncs" at the top of your awk script.