compile displays the output buffer using display-buffer (as do most other commands, fwiw). You can see this in compile.el, which is included with Emacs. There is always a link from the help for a function to the source code. Use C-h f compile RET to show the help for this function, then click the link. compile ends by calling compile-start, which does most of the work. The part you care about looks like this:
;; Pop up the compilation buffer.
;; http://lists.gnu.org/archive/html/emacs-devel/2007-11/msg01638.html
(setq outwin (display-buffer outbuf '(nil (allow-no-window . t))))
outbuf is the buffer that holds the compilation output. It calls display-buffer to make that buffer visible to the user in some way. It saves the resulting window in outwin because it probably queries or adjusts that window in some way; it may be necessary to examine what it does with that window, but for now I’m going to ignore it.
The key here is that compile does not care about how the buffer is displayed, that is display-buffer’s job. This allows display-buffer to handle policy decisions about where buffers should show up, without having to make compile care much about the details.
Let’s look at the help for display-buffer (you should use C-h f to view it in your Emacs as well):
display-buffer is an interactive compiled Lisp function in
‘window.el’.
(display-buffer BUFFER-OR-NAME &optional ACTION FRAME)
Display BUFFER-OR-NAME in some window, without selecting it.
BUFFER-OR-NAME must be a buffer or the name of an existing
buffer. Return the window chosen for displaying BUFFER-OR-NAME,
or nil if no such window is found.
Optional argument ACTION, if non-nil, should specify a display
action. Its form is described below.
Optional argument FRAME, if non-nil, acts like an additional
ALIST entry (reusable-frames . FRAME) to the action list of ACTION,
specifying the frame(s) to search for a window that is already
displaying the buffer. See ‘display-buffer-reuse-window’.
If ACTION is non-nil, it should have the form (FUNCTION . ALIST),
where FUNCTION is either a function or a list of functions, and
ALIST is an arbitrary association list (alist).
Each such FUNCTION should accept two arguments: the buffer to
display and an alist. Based on those arguments, it should
display the buffer and return the window. If the caller is
prepared to handle the case of not displaying the buffer
and returning nil from ‘display-buffer’ it should pass
(allow-no-window . t) as an element of the ALIST.
The ‘display-buffer’ function builds a function list and an alist
by combining the functions and alists specified in
‘display-buffer-overriding-action’, ‘display-buffer-alist’, the
ACTION argument, ‘display-buffer-base-action’, and
‘display-buffer-fallback-action’ (in order). Then it calls each
function in the combined function list in turn, passing the
buffer as the first argument and the combined alist as the second
argument, until one of the functions returns non-nil.
If ACTION is nil, the function list and the alist are built using
only the other variables mentioned above.
Yea, it’s pretty long (I didn’t even include most of it). There are a lot of things you can tweak and customize! Basically what it does is build a list of functions and call them each in turn until one of them has done the job. Most of these functions handle special cases, and return nil in the general case. The last function in the list should be one that handles the general case with some default action. You could add a special case for them to handle, or you could change what that final function is. Just be aware that doing the latter will change the behavior of everything calling display-buffer, which is a lot of things.
The content of the ACTION argument is also passed to each of these functions, and the content also changes their behavior. There are about half a dozen things you can put in there, and I’m not going to duplicate their documentation here.
Of the four variables mentioned in the documentation above, three default to nil. That is, they are places for the user to add customizations, rather than a place to put the default Emacs behavior. The final variable, display-buffer-fallback-action, is what implements the default Emacs behavior. On my system it has this value:
((display-buffer--maybe-same-window
display-buffer-reuse-window
display-buffer--maybe-pop-up-frame-or-window
display-buffer-in-previous-window
display-buffer-use-some-window
display-buffer-pop-up-frame))
Note however that I’m using a computer with an old version of Emacs today, so yours might be different. You can check with C-h v.
You can check the help for each of these functions to see exactly what they do (again, using C-h f), but the names are fairly clear. The final one in the list is the ultimate fallback, since it cannot fail except in quite exceptional circumstances.
The solution to your problem will depend greatly on how you have already customized Emacs, the details of your system (such as how many lines tall your Emacs frames can be, and so on), and on exactly what you want Emacs to do. As such I cannot offer very specific advice.
If you only want to change the behavior of compile, and not of other commands like grep, then I suggest adding an entry to display-buffer-alist. This alist contains entries whose first element is a regular expression that is matched against buffer names, so you can easily target just the *compile* buffer.
On the other hand, if you want to change the behavior of many commands then you should look at the functions listed in display-buffer-fallback-action. Each of these has customizable options you can tweak, such as how wide or tall a window must be before it can be split in half.
If all else fails, you could write a completely new function and add it to display-buffer-overriding-action. You could make it handle any cases you care about in whatever way you prefer. This can allow you to get the behavior you desire without needing to understand the plethora of existing options and behaviors.
Edit: something that might help you debug this is to trace all of the functions involved:
(dolist (action (car display-buffer-fallback-action))
(trace-function action))
(trace-function 'display-buffer)
You’ll end up with a trace buffer that looks something like this:
======================================================================
1 -> (display-buffer #<buffer *grep*> (nil (allow-no-window . t)))
| 2 -> (display-buffer--maybe-same-window #<buffer *grep*> ((allow-no-window . t)))
| 2 <- display-buffer--maybe-same-window: nil
| 2 -> (display-buffer-reuse-window #<buffer *grep*> ((allow-no-window . t)))
| 2 <- display-buffer-reuse-window: nil
| 2 -> (display-buffer--maybe-pop-up-frame-or-window #<buffer *grep*> ((allow-no-window . t)))
| 2 <- display-buffer--maybe-pop-up-frame-or-window: #<window 22 on *grep*>
1 <- display-buffer: #<window 22 on *grep*>
======================================================================
1 -> (display-buffer #<buffer *trace-output*> nil 0)
| 2 -> (display-buffer--maybe-same-window #<buffer *trace-output*> ((reusable-frames . 0)))
| 2 <- display-buffer--maybe-same-window: nil
| 2 -> (display-buffer-reuse-window #<buffer *trace-output*> ((reusable-frames . 0)))
| 2 <- display-buffer-reuse-window: nil
| 2 -> (display-buffer--maybe-pop-up-frame-or-window #<buffer *trace-output*> ((reusable-frames . 0)))
| 2 <- display-buffer--maybe-pop-up-frame-or-window: nil
| 2 -> (display-buffer-in-previous-window #<buffer *trace-output*> ((reusable-frames . 0)))
| 2 <- display-buffer-in-previous-window: nil
| 2 -> (display-buffer-use-some-window #<buffer *trace-output*> ((reusable-frames . 0)))
| 2 <- display-buffer-use-some-window: #<window 22 on *trace-output*>
1 <- display-buffer: #<window 22 on *trace-output*>
You can see here that it first displayed the *grep* buffer, followed by the *trace-output* buffer. Although the trace doesn’t show precisely what these functions did, it is easy to guess. In the first case, display-buffer--maybe-pop-up-frame-or-window returned a window, so it must have split the frame into two windows. When it came time to display the *trace-output* buffer though, display-buffer--maybe-pop-up-frame-or-window appears to have declined to split the frame again and it instead returned nil. Eventually display-buffer-use-some-window picked an existing window to display the buffer in, and it picked the one same one that had just been created.
Of course you can trace more low–level functions as well, to get more detail about what is going on. Reading the source of the action functions will show you what you can trace.
