43

I have trouble understanding a weird behavior: vi seems to add a newline (ASCII: LF, as it is a Unix (AIX) system) at the end of the file, when I did NOT specifically type it.

I edit the file as such in vi (taking care to not input a newline at the end):

# vi foo   ## Which I will finish on the char "9" and not input a last newline, then `:wq`
123456789
123456789
123456789
123456789
~
~
  ## When I save, the cursor is just above the last "9", and no newline was added.

I expect vi to save it "as is", so to have 39 bytes: 10 ASCII characters on each of the first three lines (numbers 1 to 9, followed by a newline (LF on my system)) and only 9 on the last line (characters 1 to 9, no terminating newline/LF).

But it appears when I save it it is 40 bytes (instead of 39), and od shows a terminating LF:

# wc foo
       4       4      40 foo  ## I expected 39 here! as I didn't add the last newline
# od -a toto
0000000    1   2   3   4   5   6   7   8   9  lf   1   2   3   4   5   6
0000020    7   8   9  lf   1   2   3   4   5   6   7   8   9  lf   1   2
0000040    3   4   5   6   7   8   9  lf
0000050
     ## An "lf" terminates the file?? Did vi add it silently?

If I create the file with a printf doing exactly what I did inside vi, it works as expected:

# ## I create a file with NO newline at the end:
# printf "123456789\n123456789\n123456789\n123456789" > foo2
# wc foo2  ## This one is as expected: 39 bytes, exactly as I was trying to do above with vi.
       3       4      39 foo  ## As expected, as I didn't add the last newline

  ## Note that for wc, there are only three lines!
  ## (So wc -l doesn't count lines; it counts the [newline] chars... Which is rather odd.)

# root@SPU0WMY1:~  ## od -a foo2
0000000    1   2   3   4   5   6   7   8   9  lf   1   2   3   4   5   6
0000020    7   8   9  lf   1   2   3   4   5   6   7   8   9  lf   1   2
0000040    3   4   5   6   7   8   9
0000047                                ## As expected, no added LF.

Both files (foo (40 characters) and foo2 (39 characters) appear exactly the same if I re-open them with vi...

And if I open foo2 (39 characters, no terminating newline) in vi and just do :wq without editing it whatsoever, it says it writes 40 chars, and the linefeed appears!

I can't have access to a more recent vi (I do this on AIX, vi (not Vim) version 3.10 I think? (no "-version" or other means of knowing it)).

# strings /usr/bin/vi | grep -i 'version.*[0-9]'
@(#) Version 3.10

Is it normal for vi (and perhaps not in more recent version? Or Vim?) to silently add a newline at the end of a file? (I thought the ~ indicated that the previous line did NOT end with a newline.)

--

Edit: some additional updates and a bit of a summary, with a big thanks to the answers below :

  • vi silently add a trailing newline at the moment it writes a file that lacked it (unless file is empty).

  • it only does so at the writing time! (ie, until you :w, you can use :e to verify that the file is still as you openened it... (ie: it still shows "filename" [Last line is not complete] N line, M character). When you save, a newline is silently added, without a specific warning (it does say how many bytes it saves, but this is in most cases not enough to know a newline was added) (thanks to @jiliagre for talking to me about the opening vi message, it helped me to find a way to know when the change really occurs)

  • This (silent correction) is POSIX behavior! (see @barefoot-io answer for references)

  • Just for completeness, which version of AIX (full version). – EightBitTony Feb 17 '16 at 15:52
  • 2
    I'm not aware of AIX's vi having this option -- appears vim-only – Jeff Schaller Feb 17 '16 at 15:54
  • 1
    @JeffSchaller : thx for the link. Unfortunately native vi doesn't have ":set noeol" nor even the -b option to open in binary mode... – Olivier Dulac Feb 17 '16 at 16:27
  • There's one special case where vi adds no newline: if the file is completely empty. – Thomas Dickey Feb 17 '16 at 20:53
  • 1
    You might be able to get vi version or at least a clue about its origin by running the :ve command. – jlliagre Feb 17 '16 at 21:00
  • On AIX, that gives the same information that OP got using strings. The behavior is well-known to vi-users. – Thomas Dickey Feb 17 '16 at 21:12
  • @ThomasDickey Yes, I'm well aware about the behavior. I was commenting about the OP stating '(no "-version" or other means of knowing it)'. It looks like IBM used SVR3 sources but removed "SVR" from the version string. – jlliagre Feb 18 '16 at 09:17
  • OP was unaware of the POSIX documentation, which lists the ve (version) command for ex. I was commenting on the behavior of newline, which OP also was unaware. A good manual would help, but AIX manual pages are not helpful to OP. – Thomas Dickey Feb 18 '16 at 09:37
  • 1
    @ThomasDickey Indeed. For some reason, IBM stripped down the ex manual page where the :ver command is normally documented. – jlliagre Feb 18 '16 at 11:19
  • @jiliagre: :ve indeed works! (but AIX man vi is lacking this information). Thanks.. in vi, :ve returns: Version 3.10 – Olivier Dulac Feb 18 '16 at 16:10
  • This is needed for POSIX compliance as far as I know. Text files must end in a newline. – bwDraco Feb 18 '16 at 17:24
  • Noting the comment that AIX vi is based on SVr3, the comment that IBM "removed" material from the document needs some verifiable source showing the unmodified SVr3 documentation to provide material for discussion. – Thomas Dickey Feb 18 '16 at 23:33
  • Please don't edit the answer to the question into the question itself. It makes it very confusing for readers. Try reading your question as if you didn't know the history -- you state the reason, then ask the very question that this answers. – Barmar Feb 24 '16 at 18:30
  • @Barmar: the thing is, that addition was starting with a Edit : thanks to ... and ... answers, that I can summarize as:, which got lost ?! (maybe because it started with a bracket?)... thanks for pointing it out. I'll try to re-edit it, and move the whole thing to the bottom of the question (or maybe remove it entirely) – Olivier Dulac Feb 24 '16 at 18:33
  • There's generally no need to put the answer into the question, that's the purpose of accepting an answer. – Barmar Feb 24 '16 at 18:37
  • @Barmar: I did not put the answer into the question: I added precision about the behaviour i saw, and as there was explanations for them at the time I added those, I pointed them out (with thanks to the answerers). I edited to move that to the end, and clarified the edit (I hope ^^) – Olivier Dulac Feb 24 '16 at 18:47
  • The bullet item explaining that this behavior is required by POSIX seems to be the answer to the question. The second bullet seems to be a tautology. When else could it modify the file other than at writing time? – Barmar Feb 24 '16 at 18:58
  • @Barmar: it could make the opened content of the file posix right at the opening stage... it doesn't (a several line file with no terminating newline : edit the first lines, and :e! to have it display its status: it still complains about the missing terminating newline). So the first 2 bullets are not a duplicate, they each add something different. And the first line is no substitute for the answers below, but is needed to read the rest of the edit (then the person will proceed to the real answers, abd get a lot more details and explanation, and concrete answers instead of just behaviours) – Olivier Dulac Feb 24 '16 at 19:04
  • Any editor that has an explicit "write" command should clearly not make any modifications to the file until told to write. – Barmar Feb 24 '16 at 19:10
  • @Barmar: you are talking about the file on the filesystem, I am talking about the file's content. Notice I several time precised "if you do :e ...", which shows the editing buffer, not the disk's file content. In other words; it complains about a lack of ending newline (and even add one when writing to the file) but it happilly let me edit as many lines as I want without fixing this, and just fixes it when :w[riting] – Olivier Dulac Feb 24 '16 at 19:21

4 Answers4

54

POSIX requires this behavior, so it's not in any way unusual.

From the POSIX vi manual:

INPUT FILES

See the INPUT FILES section of the ex command for a description of the input files supported by the vi command.

Following the trail to the POSIX ex manual:

INPUT FILES

Input files shall be text files or files that would be text files except for an incomplete last line that is not longer than {LINE_MAX}-1 bytes in length and contains no NUL characters. By default, any incomplete last line shall be treated as if it had a trailing <newline>. The editing of other forms of files may optionally be allowed by ex implementations.

The OUTPUT FILES section of the vi manual also redirects to ex:

OUTPUT FILES

The output from ex shall be text files.

A pair of POSIX definitions:

3.397 Text File

A file that contains characters organized into zero or more lines. The lines do not contain NUL characters and none can exceed {LINE_MAX} bytes in length, including the <newline> character. Although POSIX.1-2008 does not distinguish between text files and binary files (see the ISO C standard), many utilities only produce predictable or meaningful output when operating on text files. The standard utilities that have such restrictions always specify "text files" in their STDIN or INPUT FILES sections.

3.206 Line

A sequence of zero or more non- <newline> characters plus a terminating <newline> character.

These definitions in the context of these manual page excerpts mean that while a conformant ex/vi implementation must accept a malformed text file if that file's only deformity is an absent final newline, when writing that file's buffer the result must be a valid text file.

While this post has referenced the 2013 edition of the POSIX standard, the relevant stipulations also appear in the much older 1997 edition.

Lastly, if you find ex's newline appension unwelcome, you will feel profoundly violated by Seventh Edition UNIX's (1979) intolerant ed. From the manual:

When reading a file, ed discards ASCII NUL characters and all characters after the last newline. It refuses to read files containing non-ASCII characters.

Barefoot IO
  • 1,946
  • thanks, that does answer my question. i will just wait a few more days in case some better answer coles up, but right now I feel you can be the accepted answer. – Olivier Dulac Feb 17 '16 at 18:28
  • Very well done on the thoroughly documented answer, straight from the specs! :) – Wildcard Feb 18 '16 at 09:10
  • 1
    @Wildcard, the behavior preceded the specs though. – jlliagre Feb 18 '16 at 09:28
  • @jlliagre, unless you have a memoir from Bill Joy or perhaps the creator of ex (don't know his name), I think POSIX specs are as good as can be expected. ;) Closest to "original source" by this point, even though it's true they began as more or less descriptions of existing functionality. – Wildcard Feb 18 '16 at 09:39
  • 3
    @Wildcard ex was co-written by Bill Joy and Chuck Alley ( http://web.cecs.pdx.edu/~kirkenda/joy84.html .) I do not question the POSIX specs and the fact current vi releases do follow it, I just state the behavior long predates it. – jlliagre Feb 18 '16 at 10:01
33

This is the expected vi behavior.

Your file has an incomplete last line so strictly speaking (i.e. according to the POSIX standard), it is not a text file but a binary file.

vi which is a text file editor, not a binary one, gracefully fixes it when you save it.

This allows other text file tools like wc, sed and the likes to provide the expected output. Note that vi isn't silent about the issue:


$ printf "one\ntwo" >file     # Create a unterminated file
$ cat file                    # Note the missing newline before the prompt
one
two$ wc -l file               # wc ignores the incomplete last line
       1 file
$ sed '' file > file1
$ cat file1                   # so does a legacy sed
one
$ PATH=$(getconf PATH) sed  '' file
one                           # while a POSIX conformant sed warns you:
sed: Missing newline at end of file file.
two
$ vi file
one
two
~
~
~                             # vi tells you too about the issue
"file" [Incomplete last line] 2 lines, 7 characters

:w

"file" 2 lines, 8 characters  # and tells it writes two lines
                              # You'll even notice it writes one more
                              # character if you are a very shrewd observer :-)
:q
$ cat file                    # the file is now valid text
one
two
$ wc -l file                  # wc reports the expected number of lines
       2 file
$ sed '' file > file1         # sed works as expected
$ cat file1
one
two

Note, to get some clues about what vi version you are running, you can use the :ve command. It shows here I'm using a legacy SVR4 one here, definitely not vim:

:ve
Version SVR4.0, Solaris 2.5.0

Apparently, yours is stating:

:ve
Version 3.10

That likely means AIX vi is based on SVR3 source code.

In any case, this behavior, and the [Incomplete last line] warning message have been in the legacy Bill Joy's vi source code since at least 1979 and AFAIK, retained in all branches created from System V source code releases, from which proprietary Unix like AIX were built.

Chronologically speaking, this behavior is then not a consequence of POSIX conformance but more a consequence of Bill Joy's original decision to be helpful with users editing bogus text files, and then, a decade later, the POSIX committee decision to keep this tolerance.

If you use ed instead of vi, you'll notice that the former is more verbose about the issue, at least if your ed is from SVR3 or newer source branch:

$ ed file
'\n' appended
8
q

Note also that an empty file is a valid text file that happens to contain zero lines. As there is then no unterminated line to fix, vi doesn't append a newline when saving the file.

jlliagre
  • 61,204
  • 1
    I believe you mistake vim for vi ;) legacy vi is very much less verbose than this... – Olivier Dulac Feb 17 '16 at 21:09
  • @OlivierDulac I am not confusing them. This test was done using the SVR4 legacy vi just like the OP does, although on a different Unix. This is not vim or another clone. Answer updated to clarify this. – jlliagre Feb 17 '16 at 22:02
  • @OlivierDulac Hmm, I just noticed you are actually the OP. It seems AIX is using an older System V branch for its vi implementation. Possibly SVR3. Are you sure there is no [Incomplete last line] message when you open the file? – jlliagre Feb 17 '16 at 22:06
  • @OlivierDulac This link seems to imply this very same message can be displayed by AIX vi implementation: http://www-01.ibm.com/support/docview.wss?uid=isg1IZ27694 – jlliagre Feb 17 '16 at 22:16
  • I ll try to see this tomorrow – Olivier Dulac Feb 17 '16 at 22:42
  • 1979 BSD vi source which was integrated in early System V was already containing the "Incomplete last line" message. I see it too in SVR2, SVR3 and SVR4 source code. It would be very odd that IBM had removed this message from their System V branch. – jlliagre Feb 17 '16 at 22:59
  • 1
    It indeed says so when opening (easily missed), but what I don't like is that it silently changes this when writing !. Ie: printf "a" > foo ; vi foo displays "foo" [Last line is not complete] 1 line, 1 character. Then I just do: :w and it says: "foo" 1 line, 2 characters , which here is easy to spot, but sorely miss a "ADDING A NEWLINE" message (I rarely know exactly the number of bytes I had when opening, and after editing...)... I +1 you for all the research, however, and thank you for those hints. The behavior still (to me) lacks some warning (silently adding a char should be NOK) – Olivier Dulac Feb 18 '16 at 16:06
  • I understand your frustration but vi is trying to help here. The road to hell is paved with good intentions! ;-) Surprisingly, the venerable ed is more talkative in the same situation, at least under Solaris, I haven't an AIX system to check. – jlliagre Feb 18 '16 at 16:14
  • 1
    your answer is more complete than the top answer now, thanks to all your research and complements (ed behaviour, version info, etc). I still wait a bit but you took precendence now, I think. The posix citation of the top answer is a good thing though. – Olivier Dulac Feb 18 '16 at 16:22
1

Text that improperly lacks the ultimate newline run through a shell while loop results in the last line being silently discarded.

$ (echo transaction 1; echo -n transaction 2) \
  | while read line; do echo $line; done
transaction 1
$ 

Ensuring that there is an ultimate newline is the right and sane and proper default. The other option involves knowing to and having the time to audit all shell code that touches text lacking the ultimate newline, or to risk loosing the last line of the text.

thrig
  • 34,938
0

I don't recall any other behaviour that a newline is added at the end of a file (using vi since the mid 80s).

The ~ indicates that a line on the screen that is not part of the text, not that the file doesn't end in a newline. ( You can get difficult to trace errors if you put a ~ on the last line of shell scripts ). If you load a short file with a newline at the end you will see the ~ yourself and disprove that your thought it indicates non-newline-ending text.

Anthon
  • 79,293
  • what surprises me is the addition of a newline... I expect vi to not add it silently, but it seems it does... I'm looking for an explanation of this attitude (the troubling fact is : I open foo2 (without trailing LF) and just :wq, it CHANGES its content... so it shows me something but saves another thing... weird, to say the least ^^ – Olivier Dulac Feb 17 '16 at 15:48
  • in its predecessor (ed) you would create lines and edit those, not by appending characters. I always thought about vi as a line oriented editor as well. But I understand your surprise. – Anthon Feb 17 '16 at 16:22