How to bulk convert all the file in a file system branch between Unix and Windows line break format?

Question

Everybody knows :-) that in Windows plain text files lines are terminated with CR+LF, and in Unix&Linux - with LF only. How can I quickly convert all my source code files from one format to another and back?

Answer 1

That depends: if the files are under version control, this could be a rather unpopular history-polluting decision. Git has the option to automagically convert line endings on check-out.

If you do not care and want to quickly convert, there are programs like fromdos/todos and dos2unix/unix2dos that do this for you. You can use find: find . -type f -name '*.php' -exec dos2unix '{}' +.

Answer 2

There are several dedicated programs, including

• dos2unix and unix2dos from the dos2unix tools
• todos and fromdos from the tofrodos tools

Simply pick the tool for the appropriate direction and pass the names of the files to convert on the command line.

---

If you don't have either, but have Linux or Cygwin:

sed -i -e 's/\r\+$//' filename             # dos|unix -> unix
sed -i -e 's/\r*$/\r/' filename            # dos|unix -> dos

If you have perl:

perl -i -pe 's/\r+$//' filename            # dos|unix -> unix
perl -i -pe 's/\r*$/\r/' filename          # dos|unix -> dos

With only POSIX tools (including BusyBox), to go from unix to dos, you'll need to pass the CR character literally in the sed command.

cr=$(echo | tr '\n' '\r')
sed -e "s/$cr*\$/$cr/" <filename >filename.dos
mv filename.dos filename

In the other direction, you can simply delete all CRs:

tr -d '\r' <filename >filename.dos
mv filename.dos filename

---

You can use wildcards to convert many files in the same directory at once, e.g.

sed -i -e 's/\r\+$//' *.txt

To convert all files in the current directory and its subdirectories, if your shell is zsh, you can use **/, e.g.

sed -i -e 's/\r\+$//' **/*.txt

You can use **/ in bash ≥4, but you need to run shopt -s globstar first (you can put this line in your ~/.bashrc). You can use **/ in ksh93, but you need to run set -o globstar first (you can put this line in your ~/.kshrc.

If you can only use the tools that require a redirection, use a for loop.

for x in *.txt; do
  tr -d '\r' <"$x" >"$x.dos"
  mv -- "$x.dos" "$x"
done

If you don't have **/ or need more complex matching to select which files to convert, use the find command. Here's a Linux/Cygwin example which converts all files under the current directory and its subdirectories recursively, except for files called .o and under subdirectories called bin.

find -name 'bin' -type d -prune -o \
     \! -name '*.o' \
     -exec sed -i -e 's/\r\+$//' {} +

Here's a POSIX example. We tell find to start a shell that can perform the necessary redirection.

find -name 'bin' -type d -prune -o \
     \! -name '*.o' \
     -exec sh -c '
       tr -d '\r' <"$0" >"$0.dos"
       mv -- "$0.dos" "$0"
' {} \;

You can make the find method slightly faster, at the expense of more complex code, by using a loop in the shell command.

find -name 'bin' -type d -prune -o \
     \! -name '*.o' \
     -exec sh -c '
       for x; do
         tr -d '\r' <"$x" >"$x.dos"
         mv -- "$x.dos" "$x"
       done
' _ {} +

Answer 3

winscp will also auto-convert, if told so.

If your files reside all in the same directory:

DIRECTORY=/your/directory
unix2dos $DIRECTORY/*

Answer 4

Changing from ASCII text, with CRLF line terminators to ASCII text

VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Apr 10 2018 21:31:58)

---

All files in current directory

for i in *; do vi -c "set fileformat=unix | wq" "${i}"; done

---

Specific mask (all filenames in current directory starting with word "file")

for i in file*; do vi -c "set fileformat=unix | wq" "${i}"; done

---

Another mask (all filenames in current directory that contain the word "file")

for i in *file*; do vi -c "set fileformat=unix | wq" "${i}"; done

---

To convert files back to CRLF just change fileformat=unix to fileformat=dos:

for i in *file*; do vi -c "set fileformat=dos | wq" "${i}"; done

This is the quickest way if using vim to convert fileformat.

---

Alternatively sed can also be used to remove all ^M by pressing CTRL+V and then CTRL+M

for i in *; do sed -i s/^M// "$i"; done

or

for i in *; do sed -i s/^M// "${i}"; done

This will also change all CRLF files in current directory to ASCII text.