What is `^M` and how do I get rid of it?

Question

When I open the file in Vim, I see strange ^M characters.

Unfortunately, the world's favorite search engine does not do well with special characters in queries, so I'm asking here:

What is this ^M character?
How could it have got there?
How do I get rid of it?

It's a control character digraph, for more info: help digraph-table. — Eric Leschinski, Jul 01 '19 at 02:50
VS Code has an EOL option at the bottom that will auto convert an open file. — Stack Underflow, Apr 06 '20 at 20:06
@StackUnderflow when using VSCode, changing the line endings to LF when working on linux files helps. I use WSL ...A Windows file like $profile has no problem with Linux file ending LF. Maybe Win beats Linux here. — Timo, Nov 14 '20 at 08:37
My situation is, I was developing my project on Windows, later I ran a Linux virtual machine, and by sharing folders, I directly opened the project that was originally on Windows in the Linux virtual machine. When I ran git status, a large number of files were listed, and each line of these files ended with ^M. I solved this issue with the following two commands. — linjiejun, Feb 18 '24 at 07:35
git status -s | grep "^ M" | awk '{print $2}' | xargs dos2unix — linjiejun, Feb 18 '24 at 07:35
git status -s | grep "^ M" | awk '{print $2}' | xargs git add — linjiejun, Feb 18 '24 at 07:35

score 231 · Accepted Answer · edited Feb 20 '12 at 02:58

231

The ^M is a carriage-return character. If you see this, you're probably looking at a file that originated in the DOS/Windows world, where an end-of-line is marked by a carriage return/newline pair, whereas in the Unix world, end-of-line is marked by a single newline.

Read this article for more detail, and also the Wikipedia entry for newline.

This article discusses how to set up vim to transparently edit files with different end-of-line markers.

If you have a file with ^M at the end of some lines and you want to get rid of them, use this in Vim:

:s/^M$//

(Press Ctrl+V Ctrl+M to insert that ^M.)

edited Feb 20 '12 at 02:58

Gilles 'SO- stop being evil'

829,060

answered Feb 17 '12 at 23:14

larsks

34,737

28

Try :%s/^M/\r/g instead to remove ^M and replace ^M with newline character \r. Without %, the command applies for current line only. And I came across some examples where ^M is not at end of line, such as The first line.^MThe second line. – George Apr 14 '15 at 04:29
4

Or if you don't want loads of line breaks you could just do :%s/^M/ – carefulnow1 Nov 21 '16 at 08:52
2

If it's just a carriage return by itself, that might be the classic (pre-Unix) Macintosh line break. Even some newer programs like Excel 2007 for Mac do that for some reason. – sudo Apr 18 '17 at 17:22
@larsks But https://www.altap.cz/salamander/help/salamand/appendix_txtfiles/ says that ^M is used in UNIX – vivex Feb 16 '18 at 07:23
The article to which you have linked says that \n is used in Unix, which is correct. That is ASCII code 10, otherwise known as ^J. – larsks Feb 16 '18 at 21:05
8

Not sure if it's something in my vim config, but I have to type Ctrl+Q, then Ctrl+M to get that character to generate! – Brad Parks Jan 14 '20 at 19:59
I seem to get this character when I enter it in AWS lifecycle configurations. How do I get rid of it considering I am entering code in the cloud? – Luv May 19 '20 at 08:44
1

For newbies, the s means substitute, after the first slash follows the pattern which says ^M as the last character($). Substitute with nothing as seen after the second slash. Here ^ is not the beginning of the line, but part of the special char M. – Timo Nov 14 '20 at 08:41
Debian (and possibly other dists) uses this character too, at least for /var/log/apt/term.log. – Christoph Sep 08 '21 at 23:48
for what it's worth, ctrl-q is required if you have the ms windows configuration that makes ctrl-v paste – Random832 Sep 11 '21 at 11:11
1

@BradParks you are probably using (sourcing) mswin.vim. It's the default on Windows. – Cristian Ciupitu Nov 22 '21 at 17:50
This answer seems incorrect - :s/^M$// searches for a line beginning and ending with M. What is displayed in VIM as ^M is actually a carriage return \r. To remove all occurrences the command is: :%s/\r//. – Marc Jan 13 '23 at 19:12
I sometimes meet these trailing carriage returns in some text files even if the file was (correctly) loaded by VIM in DOS mode. The problem seems to be, that there's either a lone CR at the end of the line (Macintosh line break as @sudo has suggested) or better yet, a sequence of CR CR LF, i.e. there's an extra CR before a DOS line break. How does this come up? Well apparently some oddball broken software can produce this. Also note that "CR only" is the canonical line break encoding in serial terminals, and captures can contain this. I just got this from Whois for Windows by Sysinternals. – frr Oct 14 '23 at 19:01
E486: Pattern not found: ^M$ – Matthaeus Gaius Caesar Nov 11 '23 at 20:04

score 111 · Answer 2 · edited Nov 13 '17 at 16:01

111

A simpler way to do this is to use the following command:

dos2unix filename

This command works with path patterns as well, Eg

dos2unix path/name*

If it doesn't work, try using different mode:

dos2unix -c mac filename

-c Set conversion mode. Where CONVMODE is one of: ascii, 7bit, iso, mac with ascii being the default.

edited Nov 13 '17 at 16:01

Devaldo

3

answered Feb 23 '12 at 17:30

AnonGeek

1,219
1
7
4

2

+1 for mentioning about the -c flag default value and other options. At first try, the default didn't have the expected results. With dos2unix -c max it worked. dos2unix: converting file launch_me.sh to Unix format... – Johnny C Dec 19 '19 at 16:43
Amazing little utility. For ubuntu just apt-get install dos2unix. If you need to use it recursively you can refer to this solution here: https://stackoverflow.com/questions/11929461/how-can-i-run-dos2unix-on-an-entire-directory I foolishly ran it on every directory manually and only thought to look up after the fact. – fIwJlxSzApHEZIl Jul 22 '20 at 05:36

Stryker · Answer 3 · 2018-08-29T20:05:51.157

109

This worked for me

:e ++ff=dos

The :e ++ff=dos command tells Vim to read the file again, forcing dos file format. Vim will remove CRLF and LF-only line endings, leaving only the text of each line in the buffer.

then

:set ff=unix

and finally

:wq

edited Aug 29 '18 at 20:05

answered Dec 06 '16 at 20:43

Stryker

1,299
1
10
5

13

thanks! this worked for me but the accepted solution didnt – 3pitt May 30 '18 at 18:03
1

Does ff refers to file format? What about ++ff? – KcFnMi Aug 23 '18 at 18:10
1

The :e ++ff=dos command tells Vim to read the file again, forcing dos file format. Vim will remove CRLF and LF-only line endings, leaving only the text of each line in the buffer. – Stryker Aug 29 '18 at 20:05
1

To learn about ff, execute ":help ff" – sqqqrly Jan 22 '20 at 15:03
simple and effective. I am not sure why cat file1 >> file2 is introducing dos line feeds when file 1 didn't have it in the first place – abacusreader Apr 23 '20 at 12:09
great solution thank you – Sérgio Jul 08 '21 at 23:14

score 49 · Answer 4 · answered Feb 18 '12 at 03:14

49

Most UNIX operating systems have a utility called dos2unix that will convert the CRLF to LF. The other answers cover the "what are they" question.

answered Feb 18 '12 at 03:14

Aaron Brown

1,245

score 22 · Answer 5 · answered Mar 13 '20 at 09:52

22

Sed in-place solution without needing to type the special character (you can copy this and it works):

sed -i -e "s/\r//g" filename

Explanation:

-i: in-place
-e: regular expression
\r: escaped carriage return
/g: replace globally

answered Mar 13 '20 at 09:52

dreamflasher

497

The questioner was already editing the file with VIM, note. And there has been an in-place sed answer here since 2012. – JdeBP Mar 13 '20 at 10:50
Yeah, but a good, working solution was missing. The existing sed solution doesn't work out-of-the-box. – dreamflasher Mar 14 '20 at 11:11
Dale Hagglund's in-place sed most definitely does work out of the box. – JdeBP Mar 16 '20 at 10:09
1

Well, not for me. I'm on Ubuntu LTS 18.04. – dreamflasher Mar 16 '20 at 11:42
Make sure that you read the whole answer. – JdeBP Mar 16 '20 at 13:18
3

This helped me out editing a file on Alpine Linux from a Windows desktop that would count Ctrl M as a line return, making the above answers ineffective. – zacran Aug 13 '20 at 23:26
created a file ending with -e but worked – Dorian Feb 21 '21 at 20:48

score 13 · Answer 6 · edited Feb 04 '18 at 09:19

13

Another way to get rid of carriage returns is with the tr command.

I have a small script that look like this

#!/bin/sh
tmpfile=$(mktemp)
tr -d '\r' <"$1" >"$tmpfile"
mv "$tmpfile" "$1"

edited Feb 04 '18 at 09:19

Kusalananda

333,661

answered Feb 18 '12 at 21:27

Johan

4,583

score 9 · Answer 7 · answered Feb 18 '12 at 05:07

9

You can clean this up with sed:

sed -e 's/^M$//' < infile > outfile

The trick is how to enter the carriage-return properly. Generally, you need to type C-v C-m to enter a literal carriage return. You can also have sed work in place with

sed -i.bak -e 's/^M$//' infile

answered Feb 18 '12 at 05:07

Dale Hagglund

824

doesn't work (macOS) – Dorian Feb 21 '21 at 20:46
@Dorian Which part? The basic structure of the sed script is almost certain to work, but entering the literal CR (^M) depends on which terminal program you're using, and I don't know anything about Mac's in that regard. – Dale Hagglund Feb 22 '21 at 04:59
@Dorian To figure out what's not working, start by replacing the ^M with, eg, a capital A, and see if the sed command strips a final A from its input lines. If no, you're getting your basic sed syntax wrong somehow; if yes, you're having trouble correctly typing the single CR character into the shell. – Dale Hagglund Feb 22 '21 at 05:02
sorry I don't have the file to test it anymore, not sure, maybe it's because I use fish, maybe sed is different on macOS, etc. – Dorian Feb 22 '21 at 09:59
@Dorian No idea about fish, but I'm pretty sure you could create your own test file of one or two lines. – Dale Hagglund Feb 22 '21 at 10:39
@Dorian See also Dream Flasher's solution with a less tricky way to type CR – Dale Hagglund Feb 22 '21 at 10:42
Sed is usually different on Mac, try using homebrew to install gnu-sed and use the command gsed. – nathanfranke Jun 11 '21 at 23:18

Prashant Kanse · Answer 8 · 2016-01-28T07:25:03.397

5

What is this ^M?
The ^M is a carriage-return character. If you see this, you're probably looking at a file that originated in the DOS/Windows world, where an end-of-line is marked by a carriage return/newline pair, whereas in the Unix world, end-of-line is marked by a single newline.

How could it have got there?
When there is change in file format.

How do I get rid of it?
open your file with

vim -b FILE_PATH

save it with following command

:%s/^M//g

edited Jan 28 '16 at 07:25

answered Nov 21 '15 at 16:45

Prashant Kanse

167

You've got a typo in open your filr with. – Mateusz Piotrowski Jan 27 '16 at 20:48
6

This answer does not add anything to the other answers. The first paragraph is an almost verbatim copy from the accepted answer. The given code will not save anything, but just remove all carriage return characters from all lines. And I am not sure how opening the file in binary mode will help here. – Dubu Jan 28 '16 at 07:58

score 3 · Answer 9 · answered Apr 17 '16 at 05:06

3

You can use Vim in Ex mode:

ex -bsc '%s/\r//|x' file

-b binary mode
% select all lines
s substitute
\r carriage return
x save and close

answered Apr 17 '16 at 05:06

Zombo

1
5
44
63

score 2 · Answer 10 · answered Dec 07 '16 at 08:45

2

In my case,

Nothing above worked, I had a CSV file copied to Linux machine from my mac and I used all the above commands but nothing helped but the below one

tr "\015" "\n" < inputfile > outputfile

I had a file in which ^M characters were sandwitched between lines something like below

Audi,A4,35 TFSi Premium,,CAAUA4TP^MB01BNKT6TG,TRO_WBFB_500,Trico,CARS,Audi,A4,35 TFSi Premium,,CAAUA4TP^MB01BNKTG0A,TRO_WB_T500,Trico,

answered Dec 07 '16 at 08:45

Vishwanath gowda k

624
1
6
11

mac2unix -n inputfile outputfile, or, equivalently, dos2unix -c mac -n inputfile outputfile will handle that situation. – Robin A. Meade Dec 11 '19 at 01:48

score 2 · Answer 11 · answered Jan 13 '23 at 19:14

2

I have to dissent from all the answers here - what VIM displays as ^M is the carriage return character from DOS/Windows. To remove it from all lines in VIM the command is:

:%s/\r//

Trying to remove ^M with a regex like s/^M// means you don't understand regex - ^ just means the start of the line and ^M matches any line starting with capital M.

answered Jan 13 '23 at 19:14

Marc

412

1

Note that at least two answers (including the now top one) say to enter that ^M with Ctrl+V Ctrl+M, telling the software to enter it as a literal. It's not the same as entering a circumflex and the letter M. – ilkkachu Jan 13 '23 at 19:36

score 0 · Answer 12 · answered Sep 06 '19 at 14:05

0

Add the following line to your ~/.vimrc

command! Tounix :call Preserve('1,$s/^M//')

Then when you have a file with the Windows line endings, run the command ":Tounix".

answered Sep 06 '19 at 14:05

Charlie Dalsass

193

Christoph · Answer 13 · 2021-09-11T02:31:34.403

If your file uses mixed line breaks, i. e. \r\n and \r, you can use this sed script (this is an all-in-one solution and of course, you can also use it if your file merely has \r\n or \r line breaks):

sed -i[SUFFIX] ':read; N; $!b read; s/\r\n/\n/g; s/\r/\n/g' file

-i tells sed to replace your file with the result of the script, and if you supply a SUFFIX, a backup will be created with that suffix.

You also may omit -i and redirect the output to an arbitrary file.

How the script works:

Special commands:

N: Adds a newline to the pattern space, then appends the next line of input (with any trailing \n removed) to the pattern space.
$!: Don't execute the following command on the last line.
b: Branches unconditionally to the specified label.

So when the cycle starts (we have just one cycle here), sed reads the first line of input, removes any trailing \n and places it in the pattern space, then it processes the script:

N adds a newline to the pattern space, then appends the next line of input (with any trailing \n removed) to the pattern space. This is repeated until the last line is reached, then the substitutions operate on the pattern space, which contains the whole file as a single line, which is the reason why we need the g flag for the substitutions.

The first substitution replaces each \r\n with \n. After this, the remaining ("lonesome") \rs are replaced with \n.

Why do we need the N loop?

Since sed 's/<regexp>/<replacement>/[flags]' file reads a line from the file, performs the substitution on it, prints the result, reads the next line and so on, \n cannot be used within the <regexp> part.

But if we have the whole file as a single line in the pattern space, we will be able to use \n within the <regexp> part, and this is necessary to distinguish between the sequence \r\n and "lonesome" \rs.

Example:

$ sudo file /var/log/apt/term.log
/var/log/apt/term.log: UTF-8 Unicode text, with CRLF, CR, LF line terminators, with escape sequences, with overstriking
$ sudo sed -i.bak ':read; N; $!b read; s/\r\n/\n/g; s/\r/\n/g' /var/log/apt/term.log
$ sudo file /var/log/apt/term.log
/var/log/apt/term.log: UTF-8 Unicode text, with escape sequences, with overstriking

score -3 · Answer 14 · edited Jul 10 '17 at 02:45

-3

In the past, I have seen even configuration files are not parsed properly and complain about whitespace, but if you vi and do a set list it won't show the whitespace, grep filename [[space]] will show you ^M

that's when dos2unix file helps

edited Jul 10 '17 at 02:45

andrew lorien

436

answered Mar 29 '17 at 18:57

SSS

11
2

This answer does not bring nothing new, does2unix is already mentioned at least in two answers much more older. – Rui F Ribeiro Apr 12 '19 at 14:06

What is `^M` and how do I get rid of it?

14 Answers14

Linked

Related