grep --color=auto breaks when ^M is inside colored match

Question

I have 2 similar files (dos.txt and unix.txt) with text The quick brown fox jumps\n over the lazy dog. that differ by line endings. When I search for a word at the end of line, output from dos.txt is empty:

$ grep -E 'jumps^M?$' dos.txt unix.txt

unix.txt:The quick brown fox jumps

Grep finds something but doesn't print it. Actual output from grep looks like this:

$ grep -E --color=always 'jumps^M?$' dos.txt unix.txt | cat -v
^[[35m^[[Kdos.txt^[[m^[[K^[[36m^[[K:^[[m^[[KThe ... ^[[01;31m^[[Kjumps^M^[[m^[[K
^[[35m^[[Kunix.txt^[[m^[[K^[[36m^[[K:^[[m^[[KThe ... ^[[01;31m^[[Kjumps^[[m^[[K

So it looks like the only difference is that ^M is inside colored output and it causes whole line to disappear. How can I fix this (without converting dos files using dos2unix or similar tools)?

Answer 1

After some searching for ^[[K escape sequence, reading half of a book about VT100 terminal and checking man grep I have found that setting environment variable GREP_COLORS to

GREP_COLORS=ne

Gives desired output:

$ export GREP_COLORS=ne
$ grep -E --color=always 'jumps^M?$' dos.txt unix.txt
dos.txt:The quick brown fox jumps
unix.txt:The quick brown fox jumps

$ grep -E --color=always 'jumps^M?$' dos.txt unix.txt | cat -v
^[[35mdos.txt^[[m^[[36m:^[[mThe ... ^[[01;31mjumps^M^[[m
^[[35munix.txt^[[m^[[36m:^[[mThe ... ^[[01;31mjumps^[[m

From grep man page:

ne     Boolean  value  that prevents clearing to the
       end of line using Erase in Line (EL) to Right
       (\33[K)  each  time  a  colorized  item ends.
       This is needed on terminals on  which  EL  is
       not  supported.   It  is  otherwise useful on
       terminals  for  which  the   back_color_erase
       (bce)  boolean  terminfo  capability does not
       apply, when the chosen  highlight  colors  do
       not  affect the background, or when EL is too
       slow or causes too much flicker.  The default
       is false (i.e., the capability is omitted).

In my case it works good even if I set highlight color to something that change background:

export GREP_COLORS="ne:mt=41;38"

Now the interesting question is why ^[[K produces this blank line.

Character ^M means carriage return without going to next line:

$ echo -e "start^Mend"
endrt

^[[K clears line from cursor to right and then writes rest of line:

$ echo -e "start\033[Kend"
startend

However when you put ^M before ^[[K it removes content:

$ echo -e "start^M\033[Kend"
end

After writing start cursor goes to the beginning of line then ^[[K removes everything and rest of the line is written. In case of grep output first line writes everything up to word jumps, then goes back to beginning of line ^M, writes harmless ^[[m sequence and ^J that goes to new line. This is why ^[[K after ^M clears whole line.

Answer 2

You can pipe the output through tr(1) and drop the carriage returns. tr -d '\r' will drop all carriage returns.

This works:

grep --color=always -E 'jumps' dos.txt unix.txt | tr -d '\r'

(Also works with the ^M?.)

Tip: Pipe output to less(1) whenever you suspect that the output may contain unsafe characters.