Why does awk do full buffering when reading from a pipe

Question

I'm reading from a serial port connected to a gps device sending nmea strings.

A simplified invocation to illustrate my point:

  $ awk '{ print $0 }' /dev/ttyPSC9 
  GPGGA,073651.000,6310.1043,N,01436.1539,E,1,07,1.0,340.2,M,33.3,M,,0000*56
  $GPGSA,A,3,28,22,09,27,01,19,17,,,,,,2.3,1.0,2.0*39
  $GPRMC,073651.000,A,6310.1043,N,01436.1539,E,0.42,163.42,070312,,,A*67
  GPGGA,073652.000,6310.1043,N,01436.1540,E,1,07,1.0,339.2,M,33.3,M,,0000*55
  $GPGSA,A,3,28,22,09,27,01,19,17,,,,,,2.3,1.0,2.0*39

If I instead try to read from a pipe, awk buffers the input before sending it to stdout.

$ cat /dev/ttyPSC9 | awk '{ print $0 }'
<long pause>
GPGGA,073651.000,6310.1043,N,01436.1539,E,1,07,1.0,340.2,M,33.3,M,,0000*56
$GPGSA,A,3,28,22,09,27,01,19,17,,,,,,2.3,1.0,2.0*39
$GPRMC,073651.000,A,6310.1043,N,01436.1539,E,0.42,163.42,070312,,,A*67
GPGGA,073652.000,6310.1043,N,01436.1540,E,1,07,1.0,339.2,M,33.3,M,,0000*55
$GPGSA,A,3,28,22,09,27,01,19,17,,,,,,2.3,1.0,2.0*39

How can I avoid the buffering?

Edit: Kyle Jones suggested that cat is buffering it's output but that doesn't appear to be happening:

$ strace cat /dev/ttyPSC9 | awk '{ print $0 }'
write(1, "2,"..., 2)                    = 2
read(3, "E"..., 4096)                   = 1
write(1, "E"..., 1)                     = 1
read(3, ",0"..., 4096)                  = 2

When I think about it: I thought that a program used line buffering when writing to a terminal and "regular buffering" for all other cases. Then, why is cat not buffering more? Is the serial port signalling EOF? Then why is cat not terminated?

My awk is mawk 1.2.

@jw013: Thanks for the link, a great sum up of how buffering works in bash. — Daniel Näslund, Mar 07 '12 at 08:31

score 61 · Answer 1 · edited Jul 03 '16 at 23:54

61

I know it is an old question, but a one-liner may help those who come here searching:

cat /dev/ttyPSC9 | awk '{ print $0; system("")}'

system("") does the trick, and is POSIX compliant. Non-posix systems: beware.

There exists a more specific function fflush() that does the same, but is not available in older versions of awk.

An important piece of information from the docs regarding the use of system(""):

gawk treats this use of the system() function as a special case and is smart enough not to run a shell (or other command interpreter) with the empty command. Therefore, with gawk, this idiom is not only useful, it is also efficient.

edited Jul 03 '16 at 23:54

Stephen Harris

44,540

answered Jul 21 '13 at 17:28

Shrein

719

This worked for me – redolent Jan 01 '14 at 03:20
7

My awk doesn't do anything on neither fflush() nor system(""). My gawk honored it though. – Krzysztof Jabłoński Jul 14 '17 at 11:47
This worked on Mac's awk version 20200816 from Mac OS Ventura, thank god. – Sridhar Sarnobat May 09 '23 at 18:57

score 15 · Accepted Answer · edited May 24 '17 at 20:31

15

It is likely to be buffering in awk, not cat. In the first case, awk believes it is interactive because its input and output are TTYs (even though they're different TTYs - I'm guessing that awk is not checking that). In the second, the input is a pipe so it runs non-interactively.

You will need to explicitly flush in your awk program. This is not portable, though.

For more background and details on how to flush output, read: http://www.gnu.org/software/gawk/manual/html_node/I_002fO-Functions.html

edited May 24 '17 at 20:31

dhag

15,736
4
55
65

answered Mar 07 '12 at 08:25

camh

39,069

12

Thanks for the explanation. awk -W interactive '{print $0}' appears to do the trick. The 'W interactive option is available on my awk version (mawk 1.2) but I dunno if it's a standard option. – Daniel Näslund Mar 07 '12 at 08:29
3

@dannas -W is not in the POSIX standard for awk. I'm not sure what to do if you need maximum portability. – jw013 Mar 07 '12 at 08:46
I'm accepting this answer since it explains why awk is doing full buffering in my example, instead of line buffering - it checks the if the input is a tty as well as the output. I only thought that it would check the output. – Daniel Näslund Mar 07 '12 at 08:57
@jw013: Thanks for looking up the standard. For me, I just wanted to understand why awk was doing full buffering and I think I do now. – Daniel Näslund Mar 07 '12 at 09:00
@dannas I can confirm that -W interactive is at least supported in the Ubuntu 12.04 (and presumably newer) distribution of awk, which is mawk. – Jason C Jun 11 '14 at 19:38
After much searching this answered my query as to why my pipe after awk was not getting input. I followed the link to the gawk documentation and ended up using fflush("/dev/stdout") which solved my issue. – antonkronaj Oct 28 '21 at 05:06

score 5 · Answer 3 · edited Jun 13 '22 at 10:01

5

Old topic but maybe worth to add one more solution which changes stream buffering behavior in a transparent way without some magic like system("") :-)

cat /dev/ttyPSC9 | stdbuf --output=L awk '{print $0}'

Used it myself recently to catch a D-Bus event.

gdbus monitor -y -d org.freedesktop.login1 | stdbuf -oL  grep LockedHint

edited Jun 13 '22 at 10:01

AdminBee

22,803

answered Jun 13 '22 at 09:01

robo8920

51

Thanks, this fixed my problem on RHEL 7. – Mark Booth Jan 17 '23 at 15:40

Why does awk do full buffering when reading from a pipe

3 Answers3

Linked

Related