Why was `cp` designed to silently overwrite existing files?

Question

I tested cp with the following commands:

$ ls
first.html   second.html  third.html

$ cat first.html
first

$ cat second.html
second

$ cat third.html
third

Then I copy first.html to second.html:

$ cp first.html second.html

$ cat second.html
first

The file second.html is silently overwritten without any errors. However, if I do it in a desktop GUI by dragging and dropping a file with the same name, it will be suffixed as first1.html automatically. This avoids accidentally overwriting an existing file.

Why doesn't cp follow this pattern instead of overwriting files silently?

Answer 1

The default overwrite behavior of cp is specified in POSIX.

3. If source_file is of type regular file, the following steps shall be taken:

   3.a. The behavior is unspecified if dest_file exists and was written by a previous step. Otherwise, if dest_file exists, the following steps shall be taken:

   3.a.i. If the -i option is in effect, the cp utility shall write a prompt to the standard error and read a line from the standard input. If the response is not affirmative, cp shall do nothing more with source_file and go on to any remaining files.

   3.a.ii. A file descriptor for dest_file shall be obtained by performing actions equivalent to the open() function defined in the System Interfaces volume of POSIX.1-2017 called using dest_file as the path argument, and the bitwise-inclusive OR of O_WRONLY and O_TRUNC as the oflag argument.

   3.a.iii. If the attempt to obtain a file descriptor fails and the -f option is in effect, cp shall attempt to remove the file by performing actions equivalent to the unlink() function defined in the System Interfaces volume of POSIX.1-2017 called using dest_file as the path argument. If this attempt succeeds, cp shall continue with step 3b.

When the POSIX specification was written, there already was a large number of scripts in existence, with a built-in assumption for the default overwrite behavior. Many of those scripts were designed to run without direct user presence, e.g. as cron jobs or other background tasks. Changing the behavior would have broken them. Reviewing and modifying them all to add an option to force overwriting wherever needed was probably considered a huge task with minimal benefits.

Also, the Unix command line was always designed to allow an experienced user to work efficiently, even at the expense of a hard learning curve for a beginner. When the user enters a command, the computer is to expect that the user really means it, without any second-guessing; it is the user's responsibility to be careful with potentially destructive commands.

When the original Unix was developed, the systems then had so little memory and mass storage compared to modern computers that overwrite warnings and prompts were probably seen as wasteful and unnecessary luxuries.

When the POSIX standard was being written, the precedent was firmly established, and the writers of the standard were well aware of the virtues of not breaking backwards compatibility.

Besides, as others have described, any user can add/enable those features for themselves, by using shell aliases or even by building a replacement cp command and modifying their $PATH to find the replacement before the standard system command, and get the safety net that way if desired.

But if you do so, you'll find that you are creating a hazard for yourself. If the cp command behaves one way when used interactively and another way when called from a script, you may not remember that the difference exists. On another system, you might end up being careless because you're become used to the warnings and prompts on your own system.

If the behavior in scripts will still match the POSIX standard, you're likely to get used to the prompts in interactive use, then write a script that does some mass copying - and then find you're again inadvertently overwritten something.

If you enforce prompting in scripts too, what will the command do when run in a context that has no user around, e.g. background processes or cron jobs? Will the script hang, abort, or overwrite?

Hanging or aborting means that a task that was supposed to get done automatically will not be done. Not overwriting may sometimes also cause a problem by itself: for example, it might cause old data to be processed twice by another system instead of being replaced with up-to-date data.

A large part of the power of the command line comes from the fact that once you know how to do something on the command line, you'll implicitly also know how to make it happen automatically by scripting. But that is only true if the commands you use interactively also work exactly the same when invoked in a script context. Any significant differences in behavior between interactive use and scripted use will create a sort of cognitive dissonance which is annoying to a power user.

Answer 2

cp comes from the beginning of Unix. It was there well before the Posix standard was written. Indeed: Posix just formalized the existing behavior of cp in this regard.

We're talking around Epoch (1970-01-01), when men were real men, women were real women and furry little creatures ... (I digress). In those days, adding extra code made a program bigger. That was an issue then, because the first computer that ran Unix was a PDP-7 (upgradable to 144KB RAM!). So things were small, efficient, without safety-features.

So, in those days, you had to know what you were doing, because the computer just did not have the power to prevent you from doing anything you regretted later.

(There is a nice cartoon by Zevar; search for "zevar cerveaux assiste par ordinateur" to find the evolution of the computer. Or try http://a54.idata.over-blog.com/2/07/74/62/dessins-et-bd/le-CAO-de-Zevar---reduc.jpg for as long as it exists)

For those really interested (I saw some speculation in the comments): The original cp on the first Unix was about two pages of assembler code (C came later). The relevant part was:

sys open; name1: 0; 0   " Open the input file
spa
  jmp error         " File open error
lac o17         " Why load 15 (017) into AC?
sys creat; name2: 0     " Create the output file
spa
  jmp error         " File create error

(So, a hard sys creat)

And, while we're at it: Version 2 of Unix used (code sniplet)

mode = buf[2] & 037;
if((fnew = creat(argv[2],mode)) < 0){
    stat(argv[2], buf);

which is also a hard creat without tests or safeguards. Note that the C-code for V2 Unix of cp is less than 55 lines!

Answer 3

Because these commands are also meant to be used in scripts, possibly running without any kind of human supervision, and also because there are plenty of cases where you indeed want to overwrite the target (the philosophy of the Linux shells is that the human knows what s/he is doing)

There are still a few safeguards:

• GNU cp has a -n|--no-clobber option
• if you copy several files to a single one cp will complain that the last one is not a directory.

Answer 4

Is it "do one thing at one time"?

This comment sounds like a question about a general design principle. Often, questions about these are very subjective, and we are not able to write a proper answer. Be warned that we may close questions in this case.

Sometimes we have an explanation for the original design choice, because the developer(s) have written about them. But I don't have such a nice answer for this question.

Why cp is designed this way?

The problem is Unix is over 40 years old.

If you were creating a new system now, you might make different design choices. But changing Unix would break existing scripts, as mentioned in other answers.

Why was cp designed to silently overwrite existing files?

The short answer is "I don't know" :-).

Understand that cp is only one problem. I think none of the original command programs protected against overwriting or deleting files. The shell has a similar problem when redirecting output:

$ cat first.html > second.html

This command also silently overwrites second.html.

I am interested to think how all these programs could be redesigned. It might require some extra complexity.

I think this is part of the explanation: early Unix emphasized simple implementations. For a more detailed explanation of this, see "worse is better", linked at the end of this answer.

You could change > second.html so it stops with an error, if second.html already exists. However as we mentioned, sometimes the user does want to replace an existing file. For example, she may be building up a complex command, trying several times until it does what she wants.

The user could run rm second.html first if she needs to. This might be a good compromise! It has some possible disadvantages of its own.

1. The user must type the filename twice.
2. People also get in to a lot of trouble using rm. So I would like to make rm safer as well. But how? If we make rm show each filename and ask the user to confirm, she now has to write three lines of commands instead of one. Also, if she has to do this too often, she will get into a habit and type "y" to confirm without thinking. So it could be very annoying, and it could still be dangerous.

On a modern system, I recommend installing the trash command, and using it instead of rm where possible. The introduction of Trash storage was a great idea e.g. for a single-user graphical PC.

I think it is also important to understand the limitations of the original Unix hardware - limited RAM and disk space, output displayed on slow printers as well as the system and development software.

Notice that original Unix did not have tab completion, to quickly fill in a filename for an rm command. (Also, the original Bourne shell does not have command history, e.g. like when you use the Up arrow key in bash).

With printer output, you would use line-based editor, ed. This is harder to learn than a visual text editor. You have to print some current lines, decide how you want to change them, and type an edit command.

Using > second.html is a bit like using a command in a line-editor. The effect it has depends on the current state. (If second.html already exists, its content will be discarded). If the user is not sure about the current state, she is expected to run ls or ls second.html first.

"Simple implementation" as a design principle

There is a popular interpretation of Unix design, which begins:

The design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface. Simplicity is the most important consideration in a design.

...

Gabriel argued that "Worse is better" produced more successful software than the MIT approach: As long as the initial program is basically good, it will take much less time and effort to implement initially and it will be easier to adapt to new situations. Porting software to new machines, for example, becomes far easier this way. Thus its use will spread rapidly, long before a [better] program has a chance to be developed and deployed (first-mover advantage).

https://en.wikipedia.org/wiki/Worse_is_better

Answer 5

The design of "cp" goes back to the original design of Unix. There in fact was a coherent philosophy behind the Unix design, which has been slightly less that half-jokingly been referred to as Worse-is-Better^*.

The basic idea is that keeping the code simple is actually a more important design consideration that having a perfect interface or "doing The Right Thing".

• Simplicity -- the design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface. Simplicity is the most important consideration in a design.

• Correctness -- the design must be correct in all observable aspects. It is slightly better to be simple than correct.

• Consistency -- the design must not be overly inconsistent. Consistency can be sacrificed for simplicity in some cases, but it is better to drop those parts of the design that deal with less common circumstances than to introduce either implementational complexity or inconsistency.

• Completeness -- the design must cover as many important situations as is practical. All reasonably expected cases should be covered. Completeness can be sacrificed in favor of any other quality. In fact, completeness must be sacrificed whenever implementation simplicity is jeopardized. Consistency can be sacrificed to achieve completeness if simplicity is retained; especially worthless is consistency of interface.

^{(emphasis mine)}

Remembering that this was 1970, the use case of "I want to copy this file only if it doesn't already exist" would have been a fairly rare use case for someone performing a copy. If that's what you wanted, you'd be quite capable of checking before the copy, and that can even be scripted.

As to why an OS with that design approach happened to be the one that won out over all other OS's being built at the time, the author of the essay had a theory for that as well.

A further benefit of the worse-is-better philosophy is that the programmer is conditioned to sacrifice some safety, convenience, and hassle to get good performance and modest resource use. Programs written using the New Jersey approach will work well both in small machines and large ones, and the code will be portable because it is written on top of a virus.

It is important to remember that the initial virus has to be basically good. If so, the viral spread is assured as long as it is portable. Once the virus has spread, there will be pressure to improve it, possibly by increasing its functionality closer to 90%, but users have already been conditioned to accept worse than the right thing. Therefore, the worse-is-better software first will gain acceptance, second will condition its users to expect less, and third will be improved to a point that is almost the right thing.

^{* - or what the author, but nobody else, called "The New Jersey approach"}.

Answer 6

The main reason is that a GUI is by definition interactive, while a binary like /bin/cp is just a program which can be called from all kinds of places, for example from your GUI ;-). I'd bet that even today the vast majority of calls to /bin/cp will not be from a real terminal with a user typing a shell command but rather from a HTTP server or a mail system or a NAS. A built-in protection against user errors makes complete sense in an interactive environment; less so in a simple binary. For example, your GUI will most likely call /bin/cp in the background to perform the actual operations and would have to deal with the safety questions on standard out even though it just asked the user!

Note that it was from day one close to trivial to write a safe wrapper around /bin/cp if so desired. The *nix philosophy is to provide simple building blocks for users: of these, /bin/cp is one.