44

There is a thread that talks about ls "*" not showing any files, but I actually wonder why the simple

ls *

command doesn't output anything in my terminal other than

ls: invalid option -- '|'
Try 'ls --help' for more information.

while 

ls

will list all the files in the current directories as 

 1                  ferol                      readme.txt
 2                  fichier                    sarku
 2018               GameShell                  Templates
 22223333          '-|h4k3r|-'                 test
 3                  hs_err_pid2301.log         test2
 CA.txt             important.top.secret.txt   toto.text
 CA.zip             JavaBlueJProject           tp1_inf1070
 countryInfo.txt    liendur                   'tp1_inf1070_A19(2) (1)'
 currency           liensymbolique             tp1_inf1070_A19.tar
 curreny            LOL                        Videos
 Desktop            Longueuil                 'VirtualBox VMs'
 Documents          Music                      words
 douffos            numbers                    Zip.zip
 Downloads          Pictures
 examples.desktop   Public

Any ideas as to why the globbing doesn't take effect here? I'm on Ubuntu, working in the terminal, I don't know if it makes a difference.

Thanks.

muru
  • 72,889
MaelPJ
  • 605

4 Answers4

111

When you run 

ls *

globbing takes effect as usual, and the * expands to all filenames in the current directory -- including this one:

-|h4k3r|-

That starts with a -, so ls tries to parse it as an option (like -a or -l). As | isn't actually an option recognized by ls, it complains with the error message invalid option, then exits (without listing any files).

If you want to list everything in the current folder explicitly, instead try

ls ./*

...which will prefix all filenames with ./ (so that entry will not be misinterpreted as an option), or

ls -- *

...where -- is the "delimiter indicating end of options", ie. any remaining arguments are filenames.

JigglyNaga
  • 7,886
  • 3
    @rexkogitans : "There is a thread that talks about ls "*" not showing any files" says there is already a thread explaining this and that to do so here is redundant. – Eric Towers Oct 25 '19 at 20:19
  • 1
    Why wouldn't the shell expand that to '-|h4k3r|-' with quotes? Sounds like a bug to me – Thomas Weller Oct 25 '19 at 21:04
  • 8
    @ThomasWeller: The quotes aren't part of the filename; they're only present in the ls listing in the OP because newer versions of GNU ls quote filenames on output when they contain special characters. On the other hand, if you're aware of that and are suggesting that the shell add quotes when expanding a glob, that would go against the decades-old Bourne-shell style; it's not a bug if shell programmers have been working around it for longer than you've been alive. – jwodder Oct 25 '19 at 21:08
  • 7
    @jwodder No shell programmer has ever been forced to work around such filenames. Using -- to stop a utility from parsing further arguments as options is a well established way of handling unknown arguments that are not to be considered as options. It's not a hack or a workaround IMHO, just ordinary defensive programming. – Kusalananda Oct 25 '19 at 21:13
  • 3
    @jwodder: well, then the bug is that nobody fixed that the first time it was reported as a bug. Well, from now on I'll put -rf files onto every Linux system I find :-) – Thomas Weller Oct 25 '19 at 21:13
  • 1
    @ThomasWeller Make that -r -f *, and you'll catch more interesting cases. – Kusalananda Oct 25 '19 at 21:14
  • 13
    @jwodder, adding quotes wouldn't help. They only have meaning in the shell. Even if you write ls '-|' (with quotes), ls still sees an argument starting with a dash, and complains about it not being a valid option. On the other hand, if you pass the actual quotes to ls, (as if when running ls "'-|'") then it'll look for a filename containing those quotes, and probably complain that it doesn't exist. – ilkkachu Oct 25 '19 at 21:22
  • @ThomasWeller It's been known about (and considered not-a-bug) long enough to be mentioned in the Unix Haters Handbook, among other places. That's where I first heard about it, and I was surprised I couldn't find a duplicate before answering this question. – JigglyNaga Oct 26 '19 at 09:51
  • @EricTowers Ah, I was too fast to read the question correctly. I removed my comment. – rexkogitans Oct 26 '19 at 18:41
  • @ThomasWeller Was it ever reported as a bug? As they say, linux is extremely user-friendly, but it is somewhat picky who it considers a user – Hagen von Eitzen Oct 26 '19 at 19:58
  • 3
    It's not a bug, period. It's a consequence of the shell using the least syntax possible for ordinary common use cases. – chepner Oct 27 '19 at 00:45
  • 2
    @chepner, the ls * shell code has a bug though. The correct syntax would be ls -- * or probably ls -d -- * or even printf '%s\n' * as the OP probably only wants to list the files that match the pattern as opposed to list the contents of the directories that mach the pattern. – Stéphane Chazelas Oct 28 '19 at 06:49
  • Also note that ls sorts its output in the order specified by the locate. But globbing returns the filenames in ASCII order, in which - < 0 < A < B < a < b. So the filename -|h4k3r|- would appear first in the argument list. – AndyB Oct 28 '19 at 08:01
  • @AndyB, most internationalised shells (ksh93, bash, zsh, yash at least) sort as per locale as required by POSIX. – Stéphane Chazelas Oct 28 '19 at 11:09
  • @Stéphane-Chazelas That's true. Since the ls command was interpreting -|h4k3r|- as an option, I had assumed the OP was using an older shell that sorted filenames in ASCII order. But that may not be the case at all. It seems that ls will accept options anywhere in the command line, even following other filenames. Very un-Unix-like behavior! – AndyB Nov 27 '19 at 07:42
  • @AndyB, only GNU and busybox implementations and only when $POSIXLY_CORRECT is not in the environment though. – Stéphane Chazelas Nov 27 '19 at 07:47
  • In this case, I would say that POSIX has chosen the correct behavior, and GNU the wrong one. (And Linux, by default, is wrong). – AndyB Nov 27 '19 at 08:02
4

I think the important missing conceptual leap here is where the globbing is done. ls will never do any globbing itself. If you don't give it any arguments, it assumes you meant . and lists out that directory like it would any directory you gave it on the command line.

The thing that does the globbing is the shell. The shell then passes the results as a bunch of command line arguments to ls. ls has no clue if you typed those arguments, or if they are the result of a glob expansion. And so it parses them for flags like it would any other command line arguments.

This isn't how it works on Windows, which I think is a major flaw in how Windows handles command line programs.

One other interesting demonstration here is to make a directory that has a bunch of directories in it. If you then do ls * there, the shell will give all those directories as arguments, and ls will give you a listing for each of them. This just shows that assuming . if you didn't provide an argument is a simple default, not any kind of special behavior.

If you say ./* instead of * the shell will do a glob expansion that results in everything having ./ in front of it, so ls won't interpret any of the arguments as options. If you do ls -- *, ls interprets the option -- as meaning it should cease trying to process options and treat all the rest of the arguments as files or directories it should give a listing for. So, IMHO, -- is the better choice.

The -- convention is followed by the vast majority of Unix command line utilities and is a standard feature of most argument parsing libraries written for most languages.

Omnifarious
  • 1,322
  • 1
    Is it the Windows behaviour that is a major flaw? Sure, it means that each and every program has to implement globbing itself, while in the unix way, it only has to be implemented in the shell. But doing it in the program shouldn't be much more than a library call, and doing it in the program also avoids exactly the sort of problematic stuff that filenames starting with dashes cause on unixen, and avoids the need to quote the pattern in cases like find -name "*.txt" since the program can now decide what the pattern is applied against. – ilkkachu Oct 28 '19 at 04:02
  • 2
    @ilkkachu: The windows way makes it impossible to use quoting to reference a name containing glob characters. – R.. GitHub STOP HELPING ICE Oct 28 '19 at 04:07
  • @R.., Windows filenames also can't contain the main glob characters (* and ?) or control characters (incl. tab and newline), which also simplifies processing the filenames. Most of the issues with filename handling in shell scripts come from those (and spaces, which is probably the most common case, but forbidding them might be seen as too limiting). – ilkkachu Oct 28 '19 at 04:18
  • 4
    @ilkkachu - There are Unix hacks from long ago that relied on no output sanitization by ls and the fact that some terminals could be made to simulate arbitrary user input with control sequences. I always loved that Unix had no senseless limitations on what could be in a filename that existed to support a user space parser implementation. I consider the Unix way better because I can get completely different glob behavior by switching shells if I want and no program cares. It's the correct separation of concerns. – Omnifarious Oct 28 '19 at 04:31
  • @Omnifarious, by "hacks", you mean what is now known as security vulnerabilities. – ilkkachu Oct 28 '19 at 08:19
  • @Omnifarious, the thing about different shells having different (and configurable) globbing syntax is a good point, though it shouldn't be impossible to have configuration options or updates to the globbing library used by the programs. That is, if it was a single system-wide shared library or a system call, and not something compiled statically the programs, or into libraries distributed with the programs (which I understand is somewhat common in Windows). That would lead to the possibility of different utilities handling globbing in different ways, which would indeed be a major pain. – ilkkachu Oct 28 '19 at 09:46
  • Anyway, a well-designed system should be able to prevent exactly this kind of stuff. Built from scratch, I would imagine exec*() could have flags for each option to define if they're to be interpreted as control options or targets to process (or such). Even now, shells could make sure to always pass globbed filenames with a directory, i.e. expanding *.txt the same as ./*.txt. That should take care of accidental flags, but I haven't seen it even as an option anywhere. (Of course there might be some obscure program that would break from that, so there should be a way to turn it off.) – ilkkachu Oct 28 '19 at 09:56
  • 1
    There's also a (longish) study on all the problems with filenames by David Wheeler, here: https://dwheeler.com/essays/filenames-in-shell.html – ilkkachu Oct 28 '19 at 10:16
  • @ilkkachu - It is true that those represented security vulnerabilities. And in response ls and similar tools grew output sensitization to prevent the problem from happening anymore. Though, IMHO, it was actually a problem with terminals. I don't like the solution of adding special bits to execve. The whole idea of option flags and the like is not a concern the kernel should have. It's the wrong place for that kind of information. That way leads to madness and tons of security vulnerabilities. – Omnifarious Oct 28 '19 at 16:08
0

Generally the command ls * should list all the files and directories in your current working directory. There are 2 cases that would lead to the issue you are facing.

  1. If any of the files contained in the Directory have unsupported characters, such as a directory named --2 “or -2 would lead to such an error. For example:

    # ls -la | grep 2$
drwxrwxr-x  2 test test    4096 Oct 28 14:58 --2
# ls *
ls: unrecognized option '--2'

    This is because the file name starting with - will be considered as an option to the ls command.

  2. If that’s not the case, I believe that there may be some alias set in your .bashrc file. For example, if I set the alias alias ls='ls –2', I would not be able to use the ls command as it will call for an invalid option which is -2.

muru
  • 72,889
Not Named
  • 1
  • 1
    The hpyhen - character isn't an unsupported character, but as you've noted, one does have to be careful when file/directory names start with one. See the popular answer for reasons and solutions. – Chris Davies Oct 28 '19 at 12:00
-2

It means you have a file named '-|'. Because of the way the shell (bash, ksh, ...) works, all wildcard expressions are evaluated against what is in your current directory before the command is called, so if you have a file called -|, then ls * will be called as ls file1 file2 ... -| ..., and you get this error.

As for how to delete or rename it - you can use mv './-|' xyz to rename it to xyz, for example.

j4nd3r53n
  • 715