4

I have a few examples of different ways of extracting timing information from ping -c 10 google.com. For some of these pipelines, a line of output is produces every so often, just like the output of ping. For others, the lines of output are emitted all at once at after all of them have been processed. Is there a good rule for when I'll see the first behavior and when I'll see the second?

# prints output for each line as soon as it is received
# on OS X and Linux.
ping -c 10 google.com | grep -o 'time=\S*'

# prints output for each line as soon as it is received on OS X
# but not on Linux 
# (the output of ping is slightly different so it's $8 on Linux to get the time)
ping -c 10 google.com | awk '{ print $7 }'

# waits until all input is received on OS X and Linux
ping -c 10 google.com | awk -F ':' '{ print $2 }'

# prints output for line as soon as it is received on Linux and OS X
ping -c 10 google.com | sed -e 's/^.*time=\(.*\) .*$/\1/'

# waits for the end of input on OS X and Linux
ping -c 10 google.com | grep -o 'time\S*' | sed -e 's/time=//'

# as a quick check, this prints each line of output immediately 
# on OS X and Linux
ping -c 10 google.com | sed -e 's/time=//'

After looking around for a little bit, this seems to be just an issue of line buffering and some of the standard utilities behaving differently when used interactively vs non-interactively.

Greg Nisbet
  • 3,076
  • 1
    Which Linux and awk? Both the awk commands print output immediately for me with GNU awk (4.0.1, Ubuntu 14.04; 4.1.3, Arch). – muru Dec 12 '15 at 07:50

2 Answers2

8

It's about how buffering is handled with those programs.

If you want grep to output piped data immediately, use it with option --line-buffered.

ping -c 10 google.com | grep --line-buffered -o 'time\S*' | sed -e 's/time=//'

If you want awk to output piped data immediately, you can use option -W interactive.

ping -c 10 google.com | awk -W interactive '{ print $7 }'

You should read man pages for those apps to find out more.

StefanR
  • 1,392
3

Such buffering is done by the pipe when the involved descriptor is not a tty. Some commands have specific options to deal with this and force an interactive behavior.

Generic way to dealing with this

You can use stdbuf from GNU Coreutils. It manages the buffering of stdin, stdout, and stderr. You can completely disable it by setting it to 0, or use a line buffering by setting it to L :

stdbuf -i0 -o0 -e0 command
stdbuf -oL ping example.com

This command works by setting setvbuf() through a LD_PRELOAD hack. If the called command forcingly calls this function again, stdbuf may fail to disable buffering.

You can also use the unbuffer command to achieve basically the same result. This command is related to expect and may not be installed by default. It can be used quite basically as follow :

unbuffer command

Command-specific options

Regarding command-specific buffering, here is how to disable it :

  • GNU grep option do disable buffering: --line-buffered
  • GNU sed option do disable buffering: -u
  • GNU awk does not buffer when there is no controlling tty.
  • mawk, the default awk for - at least - Debian/Ubuntu, does output buffering. mawk may not work with stdbuf and provides a -Winteractive option to disable buffering.
Uriel
  • 1,383
  • 11
  • 15