80

I think I may be overlooking a relatively fundamental point regarding shell. Output from the ls command by default separates output with newlines, but the shell displays the output on a single line.

Can anyone explain this to me? I had always presumed that the output was simply separated by spaces, but now that I see the output separated by newlines, I would expect the output to be displaying on separate lines.

Example:

cpoweradm@debian:~/lpi103-4$ ls text*
text1  text2  text3

od shows that the output is separated by newlines:

cpoweradm@debian:~/lpi103-4$ ls text* | od -c
0000000   t   e   x   t   1  \n   t   e   x   t   2  \n   t   e   x   t
0000020   3  \n
0000022

If newlines are present, then why doesn't the output display as:

text1 
text2
text3
theconnorpower
  • 1,019

2 Answers2

94

When you pipe the output, ls acts differently.

This fact is hidden away in the info documentation:

If standard output is a terminal, the output is in columns (sorted vertically) and control characters are output as question marks; otherwise, the output is listed one per line and control characters are output as-is.

To prove it, try running

ls

and then

ls | less

This means that if you want the output to be guaranteed to be one file per line, regardless of whether it is being piped or redirected, you have to run

ls -1

(-1 is the number one)

Or, you can force ls | less to output in columns by running

ls -C

(-C is a capital C)

Mikel
  • 57,299
  • 15
  • 134
  • 153
  • So it does... Sneaky. Is this a general principle in UNIX that can be summarized succinctly or rather an implementation detail specific to ls? – theconnorpower Apr 01 '11 at 02:45
  • 8
    @theconnorpower: it's pretty much specific to ls. It is useful, but is clearly inconsistent and surprising. But note that some commands that produce colored output will strip the colors out when being piped too. – Mikel Apr 01 '11 at 02:46
  • 7
    @theconnorpower: Trivia: The inventors of Unix went on to write Plan9. In Plan9, ls always prints one per line, and lc always prints in columns. – Mikel Apr 01 '11 at 02:49
  • 2
    @theconnorpower: There are also programs that read the size of the terminal, and adjust their output accordingly, for example on Debian dpkg -l will use the whole width of the screen, but if it's printing to a pipe, it assumes the terminal is 80 columns wide, and abbreviates the output to make it fit if necessary. – Mikel Apr 01 '11 at 02:52
  • 1
    @Mikel Interesting to hear the ls/lc differences in Plan9. Thanks for the detailed answer. – theconnorpower Apr 01 '11 at 03:02
  • @Mikel Of course, you can always create some aliases, using the mentioned options, to mimic the plan 9 behaviour. @theconnorpower Many of these are GNU tools enhancements. – Keith Apr 01 '11 at 06:01
  • 2
    How can a program determine if its output is redirected to a file or whether it is going to a shell? – user2820379 Dec 30 '14 at 06:50
  • I find with -C I also need to use -w because some times my file names at too large for the default size – kdubs Nov 29 '18 at 11:19
  • Here is the explanation of -1 flag of ls from man: Force output to be one entry per line. This is the default when output is not to a terminal. – Timo Feb 27 '24 at 12:51
11

Your discovery highlights the primary reason why parsing the output of ls is always a bad idea. See Greg's wiki for a full explanation.

Think of your problem in reverse. You noticed that ls sometimes does and sometimes doesn't print newlines between it's output. For use in scripts or when forced by the -1 flag, it does. One newline at the end of each file. What there is no guarantee that each newline represents a new file name. In fact, if a filename contains a newline itself, the output of ls will be absolutly un-parsable. Consider these filenames:

file1
file2\nfile3
file4

When you ls -1 a directory with that in it, you would get something that looked like this:

file1
file2
file3
file4

Would you not naturally asume there were four files? So would any scripts that parse the output of ls. In reality there are three files, one of the with a tricky name, but you would not be able to figure that out from the output of ls.*

* Unless you were using the -l flag and noticed the output was borked, but your scripts would still choke.

Caleb
  • 70,105
  • 4
    If you really have to parse the output of ls, the -b option can help. It turns the newline into \n, etc. – Mikel Apr 10 '12 at 15:05
  • You say parsing it is a bad idea. Any alternatives? – Bluu Jan 21 '21 at 19:00
  • @Bluu Use shell globbing, or find -exec, or tell whatever you are using to use null terminators instead of newlines (many tools other than ls support this), or any of dozens of other options. There is tons written on this already or ask a specific question about what's needed. – Caleb Jan 26 '21 at 19:28