Does vi silently add a newline (LF) at the end of file?

Question

I have trouble understanding a weird behavior: vi seems to add a newline (ASCII: LF, as it is a Unix (AIX) system) at the end of the file, when I did NOT specifically type it.

I edit the file as such in vi (taking care to not input a newline at the end):

# vi foo   ## Which I will finish on the char "9" and not input a last newline, then `:wq`
123456789
123456789
123456789
123456789
~
~
  ## When I save, the cursor is just above the last "9", and no newline was added.

I expect vi to save it "as is", so to have 39 bytes: 10 ASCII characters on each of the first three lines (numbers 1 to 9, followed by a newline (LF on my system)) and only 9 on the last line (characters 1 to 9, no terminating newline/LF).

But it appears when I save it it is 40 bytes (instead of 39), and od shows a terminating LF:

# wc foo
       4       4      40 foo  ## I expected 39 here! as I didn't add the last newline
# od -a toto
0000000    1   2   3   4   5   6   7   8   9  lf   1   2   3   4   5   6
0000020    7   8   9  lf   1   2   3   4   5   6   7   8   9  lf   1   2
0000040    3   4   5   6   7   8   9  lf
0000050
     ## An "lf" terminates the file?? Did vi add it silently?

If I create the file with a printf doing exactly what I did inside vi, it works as expected:

# ## I create a file with NO newline at the end:
# printf "123456789\n123456789\n123456789\n123456789" > foo2
# wc foo2  ## This one is as expected: 39 bytes, exactly as I was trying to do above with vi.
       3       4      39 foo  ## As expected, as I didn't add the last newline

  ## Note that for wc, there are only three lines!
  ## (So wc -l doesn't count lines; it counts the [newline] chars... Which is rather odd.)

# root@SPU0WMY1:~  ## od -a foo2
0000000    1   2   3   4   5   6   7   8   9  lf   1   2   3   4   5   6
0000020    7   8   9  lf   1   2   3   4   5   6   7   8   9  lf   1   2
0000040    3   4   5   6   7   8   9
0000047                                ## As expected, no added LF.

Both files (foo (40 characters) and foo2 (39 characters) appear exactly the same if I re-open them with vi...

And if I open foo2 (39 characters, no terminating newline) in vi and just do :wq without editing it whatsoever, it says it writes 40 chars, and the linefeed appears!

I can't have access to a more recent vi (I do this on AIX, vi (not Vim) version 3.10 I think? (no "-version" or other means of knowing it)).

# strings /usr/bin/vi | grep -i 'version.*[0-9]'
@(#) Version 3.10

Is it normal for vi (and perhaps not in more recent version? Or Vim?) to silently add a newline at the end of a file? (I thought the ~ indicated that the previous line did NOT end with a newline.)

--

Edit: some additional updates and a bit of a summary, with a big thanks to the answers below :

• vi silently add a trailing newline at the moment it writes a file that lacked it (unless file is empty).

• it only does so at the writing time! (ie, until you :w, you can use :e to verify that the file is still as you openened it... (ie: it still shows "filename" [Last line is not complete] N line, M character). When you save, a newline is silently added, without a specific warning (it does say how many bytes it saves, but this is in most cases not enough to know a newline was added) (thanks to @jiliagre for talking to me about the opening vi message, it helped me to find a way to know when the change really occurs)

• This (silent correction) is POSIX behavior! (see @barefoot-io answer for references)

Answer 1

POSIX requires this behavior, so it's not in any way unusual.

From the POSIX vi manual:

INPUT FILES

See the INPUT FILES section of the ex command for a description of the input files supported by the vi command.

Following the trail to the POSIX ex manual:

INPUT FILES

Input files shall be text files or files that would be text files except for an incomplete last line that is not longer than {LINE_MAX}-1 bytes in length and contains no NUL characters. By default, any incomplete last line shall be treated as if it had a trailing <newline>. The editing of other forms of files may optionally be allowed by ex implementations.

The OUTPUT FILES section of the vi manual also redirects to ex:

OUTPUT FILES

The output from ex shall be text files.

A pair of POSIX definitions:

3.397 Text File

A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. Although POSIX.1-2008 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files. The standard utilities that have such restrictions always specify "text files" in their STDIN or INPUT FILES sections.

3.206 Line

A sequence of zero or more non- <newline> characters plus a terminating <newline> character.

These definitions in the context of these manual page excerpts mean that while a conformant ex/vi implementation must accept a malformed text file if that file's only deformity is an absent final newline, when writing that file's buffer the result must be a valid text file.

While this post has referenced the 2013 edition of the POSIX standard, the relevant stipulations also appear in the much older 1997 edition.

Lastly, if you find ex's newline appension unwelcome, you will feel profoundly violated by Seventh Edition UNIX's (1979) intolerant ed. From the manual:

When reading a file, ed discards ASCII NUL characters and all characters after the last newline. It refuses to read files containing non-ASCII characters.

Answer 2

This is the expected vi behavior.

Your file has an incomplete last line so strictly speaking (i.e. according to the POSIX standard), it is not a text file but a binary file.

vi which is a text file editor, not a binary one, gracefully fixes it when you save it.

This allows other text file tools like wc, sed and the likes to provide the expected output. Note that vi isn't silent about the issue:

$ printf "one\ntwo" >file     # Create a unterminated file
$ cat file                    # Note the missing newline before the prompt
one
two$ wc -l file               # wc ignores the incomplete last line
       1 file
$ sed '' file > file1
$ cat file1                   # so does a legacy sed
one
$ PATH=$(getconf PATH) sed  '' file
one                           # while a POSIX conformant sed warns you:
sed: Missing newline at end of file file.
two
$ vi file
one
two
~
~
~                             # vi tells you too about the issue
"file" [Incomplete last line] 2 lines, 7 characters

:w

"file" 2 lines, 8 characters  # and tells it writes two lines
                              # You'll even notice it writes one more
                              # character if you are a very shrewd observer :-)
:q
$ cat file                    # the file is now valid text
one
two
$ wc -l file                  # wc reports the expected number of lines
       2 file
$ sed '' file > file1         # sed works as expected
$ cat file1
one
two

Note, to get some clues about what vi version you are running, you can use the :ve command. It shows here I'm using a legacy SVR4 one here, definitely not vim:

:ve
Version SVR4.0, Solaris 2.5.0

Apparently, yours is stating:

:ve
Version 3.10

That likely means AIX vi is based on SVR3 source code.

In any case, this behavior, and the [Incomplete last line] warning message have been in the legacy Bill Joy's vi source code since at least 1979 and AFAIK, retained in all branches created from System V source code releases, from which proprietary Unix like AIX were built.

Chronologically speaking, this behavior is then not a consequence of POSIX conformance but more a consequence of Bill Joy's original decision to be helpful with users editing bogus text files, and then, a decade later, the POSIX committee decision to keep this tolerance.

If you use ed instead of vi, you'll notice that the former is more verbose about the issue, at least if your ed is from SVR3 or newer source branch:

$ ed file
'\n' appended
8
q

Note also that an empty file is a valid text file that happens to contain zero lines. As there is then no unterminated line to fix, vi doesn't append a newline when saving the file.

Answer 3

Text that improperly lacks the ultimate newline run through a shell while loop results in the last line being silently discarded.

$ (echo transaction 1; echo -n transaction 2) \
  | while read line; do echo $line; done
transaction 1
$ 

Ensuring that there is an ultimate newline is the right and sane and proper default. The other option involves knowing to and having the time to audit all shell code that touches text lacking the ultimate newline, or to risk loosing the last line of the text.

Answer 4

I don't recall any other behaviour that a newline is added at the end of a file (using vi since the mid 80s).

The ~ indicates that a line on the screen that is not part of the text, not that the file doesn't end in a newline. ( You can get difficult to trace errors if you put a ~ on the last line of shell scripts ). If you load a short file with a newline at the end you will see the ~ yourself and disprove that your thought it indicates non-newline-ending text.