5

In a Linux terminal (CentOS) I am using the command tail --follow=name my-rolling-file.log in order to see the logs of my application.

Sometimes in the log, there is some binary data dumped (I dump the body of a Camel Message that usually contains strings, but sometimes binary and/or special characters like Chinese in UTF-8) and when it happens my terminal gets corrupted like the pipe characters | are now ö instead.

I just guessed it is the binary data logging who can cause the problem and I am wondering if it is possible to ask the tail command to ignore special characters. I checked the man page but did not find anything there.

Currently to fix the problem I have to Ctrl-C the tailing, make a reset in the terminal and relaunch the tail command. I want to prevent these operations if possible.

If you know another command than tail, but with the same features (following rolling files) it would be acceptable as well as long it is installable and runnable under CentOS 6.5.

рüффп
  • 1,707
  • 3
    I suggest to use less my-rolling-file.log, when the file has been opened, press F (uppercase f) and you will have the same behavior as tail -f.

    Don't know if will be better treating binary data. You have to try and let us know.

     – jcbermu Apr 14 '15 at 10:34
  • Using lesscommand does not satisfy my as i have to make ctrl-c to be able to move in the history. For the binary, I got a popup asking me if i want to install new fonts for the terminal and it shows many times, the solution given with tail will be the best one for my case. Thanks anyway for your feedback. – рüффп Apr 14 '15 at 13:05

4 Answers4

14

You can translate special characters (binary data) into ordinary characters that are safe to display by piping your tail command into cat -v:

tail --follow=name my-rolling-file.log | cat -v

The -v (verbose) option of cat (also known as --show-nonprinting) displays "nonprinting" characters using the ^ and M- notations:

  • 0x00 is displayed as ^@
  • 0x01 is displayed as ^A
  • 0x02 is displayed as ^B
  • etc.
  • 0x7F is displayed as ^?
  • 0x80-0xFF are displayed as M- followed by the representation of the low order seven bits of the byte.

Or, if you prefer to discard the characters, use:

tail --follow=name my-rolling-file.log | tr -cd "\t\n -~"

tr "translates" characters; i.e., maps them.  It can be used, for example, to change lower case to upper case.  The -d option tells it to delete characters; for example, tr -d aeiou deletes all (lower case) vowels.  The -c option means complement; i.e., do all characters other than the ones I specify.  "\t\n -~" is tab, newline, and all the ordinary printable characters.

Scott - Слава Україні
  • 10,519
7

What about this,

tail --follow=name my-rolling-file.log | strings

The default for strings is that it will only output printable characters in lengths of 4 (or more), but you can change this with -n {number}.

Chris Davies
  • 116,213
  • 16
  • 160
  • 287
  • It work very well and I can choose how large the binary "strings" should be. I just created a function like this: ftail () { tail --follow=name $@ | strings -n 20 ;} and it's working perfectly. – рüффп Apr 14 '15 at 13:07
  • 2
    @ruffp You should quote the "$@" as otherwise it will break when you get filenames that contain spaces and other special characters. – Chris Davies Apr 14 '15 at 14:09
  • The short form of --follow=name is -F – Volker Siegel Apr 14 '15 at 14:21
  • @VolkerSiegel not exactly as -F is equivalent to --follow=name --retry but that should not be a problem. – рüффп Oct 17 '18 at 13:14
  • @рüффп Oh, you're right, thank you! That's quite an interesting difference. Seems like --retry makes it work even when the file does not exist at all for a while. – Volker Siegel Oct 17 '18 at 14:03
  • @VolkerSiegel yes, that's correct. I'd like to say that didn't include that in my suggestion (instead of --follow=name) because I like to keep my answers as close as possible to the asker's suggested code. (On the other hand it may be that at the time, I simply forgot.) – Chris Davies Oct 17 '18 at 14:49
3

Convert (some troublesome) characters to . with tr:

tail -f data | tr "\000-\011\013-\037\177-\377" "."
FloHimself
  • 11,492
3

Let less handle the problem, as it's good at it.

The pager less is pretty good at formating binary files for output. Also, the pager features, like scrollback and searching are often useful.
It has no direct option for following a file that grows, but it has way to provide keystrokes to "type" after startup.

The key F starts following the growing file, updating the screen when the file grows.
Prefix the key with + instead of - and use that just like an option:

Replacing tail -f:

less +F growing-file.log

To continue following the file by name, even if it's moved away or deleted, and recreated with the same name, add the option --follow-name; It modifies what the key F and +F do, it does not start following by itself:

Replacing tail -F (--follow=name):

less --follow-name +F rolling-file.log

If the file is recognized as a "binary file", less asks whether you really want to see it - to avoid the question, use the option -f (--force) too:

less --follow-name +F -f rolling-binary-file.log

(Note that the meaning of -f for less is different from the same option for tail.)

Use Ctrl+c to stop following, which allows you to scroll back or search.

  • Start following again with F.
  • Jump to start: g.
  • Jump to start: G.
  • Search: /.
  • Quit: q.

Binary output example

$ less +F /usr/bin/vim 
"/usr/bin/vim" may be a binary file.  See it anyway?

After pressing y (which you can avoid with -f) the screen shows:

 ^A^@^@^@^@^@<E5>
 ^A^@^@^@^@^@<E6>
 ^A^@^@^@^@^@^@^P^A^@^@^@^@^@^B^P^A^@^@^@^@^@8^P^A^@^@^@^@^@F^P^A^@^@^@^@^@
 <F1>^A^@^@^@^@^@^P<F1>^A^@^@^@^@^@-<F1>^A^@^@^@^@^@0<F1>^A^@^@^@^@^@i<F1>^A
 ^L(^@^@^@^@^@^@^@^@^@^@^@^@^@<A3><A5>&^@^@^@^@^@^@^@^@^@^@^@^@^@b['^@^@^@^@
 @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@@<A7>&^@^@^@^@
 <AB>&^@^@^@^@^@Ȣ&^@^@^@^@^@^L,^@^@^@^@^@^@<FF><FF><FF><FF><FF><FF><FF><FF>
  ^@^@^@^@^@^@s]&^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^_<D3>%^@^@^@^@^@̢
 <AF>&^@^@^@^@^@<CA><(^@^@^@^@^@^B^D^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
 <B1>&^@^@^@^@^@^D^D^H^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^R<B1>&^@^@
 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<A9><B3>&^@^@^@^@^@D<B8>&^@^@
 <B5>&^@^@^@^@^@^]T&^@^@^@^@^@^BL^@^@^@^@^@^@H<8A>O^@^@^@^@^@F@^@^@^@^@^@^@
 K'^@^@^@^@^@^C<C4><FF><FF>^@^@^@^@^OK'^@^@^@^@^@^C<C3><FF><FF>^@^@^@^@^TK'
 #^@^@^@^@^@^D^@^@^@^@^@^@^@<E7>v&^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
 #^@^@^@^@^@^D^@^@^@^@^@^@^@<E7>v&^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
 #^@^@^@^@^@p|#^@^@^@^@^@P~#^@^@^@^@^@^@^@^@^@^@^@^@^@^P
 #^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<90>{#^@^@^@^@^@^@^@^@^@^@^@^@^@
Waiting for data... (interrupt to abort)

Control chars are shown in carret notation like ^A (^@ is the null byte).
They are also highlighted in the terminal (which is not shown above), to make the character Control-A, ^A different from the character ^ followed by A.

Configuration of how to show binary characters

From man less:

Control  and  binary  characters  are  displayed  in  standout (reverse
video).  Each such character is displayed in caret notation if possible
(e.g.  ^A for control-A).  Caret notation is used only if inverting the
0100 bit results in a normal printable character.  Otherwise, the char‐
acter  is displayed as a hex number in angle brackets.  This format can
be changed by setting the LESSBINFMT environment variable.   LESSBINFMT
may begin with a "*" and one character to select the display attribute:
"*k" is blinking, "*d" is bold, "*u" is underlined, "*s"  is  standout,
and  "*n"  is  normal.  If LESSBINFMT does not begin with a "*", normal
attribute is assumed.  The remainder of LESSBINFMT is  a  string  which
may  include one printf-style escape sequence (a % followed by x, X, o,
d, etc.).  For example, if LESSBINFMT is  "*u[%x]",  binary  characters
are  displayed  in  underlined hexadecimal surrounded by brackets.  The
default if no LESSBINFMT is  specified  is  "*s<%02X>".   Warning:  the
result  of  expanding the character via LESSBINFMT must be less than 31
characters.

When the character set is utf-8, the LESSUTFBINFMT environment variable
acts similarly to LESSBINFMT but it applies to Unicode code points that
were successfully decoded but are unsuitable for display  (e.g.,  unas‐
signed  code  points).   Its  default  value is "<U+%04lX>".  Note that
LESSUTFBINFMT and LESSBINFMT  share  their  display  attribute  setting
("*x")  so specifying one will affect both; LESSUTFBINFMT is read after
LESSBINFMT so its setting, if any,  will  have  priority.   Problematic
octets  in  a  UTF-8  file (octets of a truncated sequence, octets of a
complete but non-shortest form  sequence,  illegal  octets,  and  stray
trailing  octets)  are displayed individually using LESSBINFMT so as to
facilitate diagnostic of how the UTF-8 file is ill-formed.
Community
  • 1
Volker Siegel
  • 17,283
  • 1
    Comprehensive answer, but less doesn't automatically update when the log file is replaced (as, for example, managed by logrotate). I believe this was part of the requirement. – Chris Davies Apr 14 '15 at 14:11
  • @roaima True - I'm about to write a question to find out whether that's possible. – Volker Siegel Apr 14 '15 at 14:12
  • That would be interesting to see. I'll look out for it - and any helpful answer. – Chris Davies Apr 14 '15 at 14:13
  • 2
    @roaima I asked at Does less have a feature like tail --follow=name – Volker Siegel Apr 14 '15 at 14:18
  • Yes, nice answer but as roaima said: --follow=name (or -F) is a requirement for me. As well another tail feature I really like is that I can make Enter to empty the terminal in order to see what has been written since that blanks. I was not able to do so with less – рüффп Apr 14 '15 at 14:40
  • @ruffp Yes, I see; Interesting point with marking up to where you have seen the output. In less, there is an option -w or --hilite-unread for something similar, but that does not work in the follow mode. (If I don't manage to make this answer fit your question, I'll try to create the right question for it ;) ) – Volker Siegel Apr 14 '15 at 16:20
  • @ruffp The other question turned up that less does have an option --follow-name that I missed somehow; It's different from tails --follow=name in that it does not start following, but modifies how following works when +F or the F key is used. See edit. (No solution yet for the "empty screen" part.) – Volker Siegel Apr 15 '15 at 00:07
  • @roaima Found it - I edited the answer. – Volker Siegel Apr 15 '15 at 00:53