8

I'm using the following statement (simplified version):

tail -f -c+1 <filename>

in order to pipe a file to a process.

What I've found, though, is that there is a number of lines at the end which are not piped.

A specific example is piping a mysql file, and stopping when the end is reached:

tail -f -c+1 mysqdump.sql | sed '/^-- Dump completed/ q0'

This doesn't work - the last line of the dump -- Dump completed [...] is not piped to sed.

My guess is that the tail -f buffer, in this case, is flushed only when it's full.

Does anybody know how can I workaround this?

=================

I've found the cause - the description is not complete (and the code doesn't exhibit the behavior).

The problem happens when piping from a compressed (lzma) file:

tail -f -c+1 <filename.lzma> | lzma -d | sed '/^-- Dump completed/ q0'

Most likely, tail is not sending the last compressed block, because it doesn't detect any new line, as the input is binary.

Marcus
  • 961
  • 2
    See e.g. http://stackoverflow.com/questions/5295430/tail-f-log-log-grep-responsetime-cut-d-f-2 for an answer. – jofel Aug 30 '12 at 13:40
  • I don't understand your means.before tail and after tail what do you do? – PersianGulf Aug 30 '12 at 18:34
  • and especially read the BashFAQ entry linked to in that stackoverflow question, http://mywiki.wooledge.org/BashFAQ/009 – cas Aug 30 '12 at 22:59

3 Answers3

12

tail -f flushes after every input line. You can confirm this with strace (or truss or whatever your unix variant provides to trace processes' system calls).

If there is an incomplete line, tail -f keeps waiting for the next newline. It's a tool designed for text files. If you need to tail a binary file (e.g. if -- Dump completed is not followed by a newline), you'll have to use a custom tool.

If you've redirected sed's output away from the terminal, it will be doing its own buffering. Try stdbuf or unbuffer.

Community
  • 1
Gilles 'SO- stop being evil'
  • 829,060
  • I was in fact executing on a binary file being remotely copied (I purposely don't want to use "ssh cat"). I worked the problem around using scp on a named pipe. – Marcus Aug 31 '12 at 08:41
  • 2
    For sed you can use -u if you don't want buffer – agate Jul 10 '18 at 16:59
4

Example using stdbuf -o0. Reference: https://superuser.com/a/1123674/126847

tail -F cpu.log | stdbuf -o0 tr '=' ' ' | stdbuf -o0 awk '$6 > 1'
JohnMudd
  • 260
1

One problem with the command invoked in the question is that sed is invoked in buffering mode.

tail -f -c+1 <filename.lzma> | lzma -d | sed '/^-- Dump completed/ q0'

You need to add the -u flag:

tail -f -c+1 <filename.lzma> | lzma -d | sed -u '/^-- Dump completed/ q0'

For more info, read about the -u switch on the sed man page.

Jeff
  • 111
  • 2