8

I have a large utf-8 text file which I frequently search with grep. Recently grep began reporting that it was a binary file. I can continue to search it with grep -a, but I was wondering what change made it decide that the file was now binary.

I have a copy from last month where the file is no longer detected as binary, but it's not practical to diff them since they differ on > 20,000 lines.

file identifies my file as

UTF-8 Unicode English text, with very long lines

How can I find the characters/lines/etc. in my file which are triggering this change?

The similar, non-duplicate question 19907 covers the possibility of NUL but grep -Pc '[\x00-\x1F]' says that I don't have NUL or any other ANSI control chaarcters.

Community
  • 1
Charles
  • 279
  • I would try this in this order: 1. Run it with strace/ltrace to check what input causes that 'binary' message 2. Check out grep's source and read it – ott-- Sep 17 '15 at 20:51
  • @muru: I'm using gnu grep, but if you have the answer for some other version I'd be interested as well. – Charles Sep 17 '15 at 21:50
  • Odd. I have a file which I know contains a nul and some Escs. I tried grepping for them. I could find the escs (\x1B), but the nul never showed up. The test given above showed 1, for the line containing Escs, but nothing for any range that didn't contain \x1B. I wouldn't trust that test. Try grep -zc . instead (should be one more than the number of nuls in your file). (Also, you might be better off using [[:cntrl:]].) – muru Sep 17 '15 at 22:10
  • Also try: sed -z 's/.*\(....\)$/\1/' foo | od -c to see a few characters before the NUL (if there is one), which might lead you to the problem. – muru Sep 17 '15 at 22:17
  • @muru: My sed doesn't have a -z option: sed: invalid option -- 'z'. – Charles Sep 17 '15 at 22:19
  • @Charles but does grep? What did grep -zc . say? – muru Sep 17 '15 at 22:26
  • Try grep -C 2 -aoP '\0' file | od -c, that should show the NUL (if present) and the surrounding lines. – terdon Sep 17 '15 at 22:41
  • @muru Note that it's impossible to pass a null byte on the command line, even if your shell appears to allow it. You can however include a null byte in a pattern with GNU grep by passing the pattern in a file (grep -f), or by using \0 or the like in Perl syntax as suggested by terdon. – Gilles 'SO- stop being evil' Sep 18 '15 at 00:03
  • @Gilles I hadn't tried to (not directly, anyway, but via [\x00] with grep -P). Or did I? – muru Sep 18 '15 at 00:05

3 Answers3

2

It appears to be the presence of the null character in the file.(displayed ^@ usually) I entered various control characters into a text file(like delete, ^?, for example), and only the null character caused grep to consider it a binary. This was only tested for grep. The less and diff commands, for instance, may have different methods. Control characters in general don't appear except in binaries. The exceptions are the whitespace characters: newline(^M), tab(^I), formfeed(^L), vertical tab(^K), and return(^J).

However, foreign characters, like arabic or chinese letters, are not standard ascii, and perhaps could be confused with control characters. Perhaps that's why it's only the null character.

You can test it out for yourself by insterting control characters into a text file using the text editor vim. Just go to insert mode, press control-v, and then the control character.

anotherguy
  • 483
  • 1
  • 5
  • 9
2

A typical modern grep implementation should only declare a file "binary" if there are nul bytes inside. Anything else should be OK.

I cannot speak for the grep implementation you use...

schily
  • 19,173
1

An encoding error according to mbrlen() also makes GNU grep 2.24 consider it as binary

E.g.:

export LC_CTYPE='en_US.UTF-8'
printf 'a\x80' | grep 'a'

because \x80 cannot be the first byte of an UTF-8 Unicode point: https://en.wikipedia.org/wiki/UTF-8#Description

This is the only other possibility besides NUL.

GNU grep source code interpretation that leads to this conclusion: What makes grep consider a file to be binary?

Community
  • 1
Ciro Santilli OurBigBook.com
  • 18,092
  • 4
  • 117
  • 102