Run command in background with foreground terminal access

Question

I am trying to create a function that can run an arbitrary command, interact with the child process (specifics omitted), and then wait for it to exit. If successful, typing run <command> will appear to behave just like a bare <command>.

If I weren't interacting with the child process I would simply write:

run() {
    "$@"
}

But because I need to interact with it while it runs, I have this more complicated setup with coproc and wait.

run() {
    exec {in}<&0 {out}>&1 {err}>&2
    { coproc "$@" 0<&$in 1>&$out 2>&$err; } 2>/dev/null
    exec {in}<&- {out}>&- {err}>&-

    # while child running:
    #     status/signal/exchange data with child process

    wait
}

^{(This is a simplification. While the coproc and all the redirections aren't really doing anything useful here that "$@" & couldn't do, I need them all in my real program.)}

The "$@" command could be anything. The function I have works with run ls and run make and the like, but it fails when I do run vim. It fails, I presume, because Vim detects that it is a background process and doesn't have terminal access, so instead of popping up an edit window it suspends itself. I want to fix it so Vim behaves normally.

How can I make coproc "$@" run in the "foreground" and the parent shell become the "background"? The "interact with child" part neither reads from nor writes to the terminal, so I don't need it to run in the foreground. I'm happy to hand over control of the tty to the coprocess.

It is important for what I'm doing that run() be in the parent process and "$@" be in its child. I can't swap those roles. But I can swap the foreground and background. (I just don't know how to.)

Note that I am not looking for a Vim-specific solution. And I would prefer to avoid pseudo-ttys. My ideal solution would work equally well when stdin and stdout are connected to a tty, to pipes, or are redirected from files:

run echo foo                               # should print "foo"
echo foo | run sed 's/foo/bar/' | cat      # should print "bar"
run vim                                    # should open vim normally

---

Why using coprocesses?

I could have written the question without coproc, with just

run() { "$@" & wait; }

I get the same behavior with just &. But in my use case I am using the FIFO coproc sets up and I thought it best not to oversimplify the question in case there's a difference between cmd & and coproc cmd.

Why avoiding ptys?

run() could be used in an automated context. If it's used in a pipeline or with redirections then there wouldn't be any terminal to emulate; setting up a pty would be a mistake.

Why not using expect?

I'm not trying to automate vim, send it any input or anything like that.

Answer 1

I've added code so that:

it works for your three examples and

the interaction comes before the wait.

interact() {
    pid=$1
    ! ps -p $pid && return
    ls -ld /proc/$pid/fd/*
    sleep 5; kill -1 $pid   # TEST SIGNAL TO PARENT
}

run() {
    exec {in}<&0 {out}>&1 {err}>&2
    { coproc "$@" 0<&$in 1>&$out 2>&$err; } 2>/dev/null
    exec {in}<&- {out}>&- {err}>&-
    { interact $! <&- >/tmp/whatever.log 2>&1& } 2>/dev/null
    fg %1 >/dev/null 2>&1
    wait 2>/dev/null
}

The fg %1 will run for all commands (change %1 as needed for concurrent jobs) and under normal circumstances one of two things will happen:

If the command exits immediately interact() will return immediately since there is nothing to do and the fg will do nothing.

If the command does not exit immediately interact() can interact (eg. send a HUP to the coprocess after 5 seconds) and the fg will put the coprocess in the foreground using the same stdin/out/err with which it was originally run (you can check this with ls -l /proc/<pid>/df).

The redirections to /dev/null in the last three commands are cosmetic. They allow run <command> to appear exactly the same as if you had run command on its own.

Answer 2

In your example code, Vim gets suspended by the kernel via a SIGTTIN signal as soon as it tries to read from the tty, or possibly set some attributes to it.

This is because the interactive shell spawns it in a different process-group without (yet) handing over the tty to that group, that is putting it “in background”. This is normal job control behavior, and the normal way to hand over the tty is to use fg. Then of course it’s the shell that goes to the background and thus gets suspended.

All this is on purpose when a shell is interactive, otherwise it would be as if you were allowed to keep typing commands at the prompt while eg editing a file with Vim.

You could easily work around that by just making your whole run function a script instead. That way, it would be executed synchronously by the interactive shell with no competing of the tty. If you do so, your own example code already does all you are asking, including concurrent interaction between your run (then a script) and the coproc.

If having it in a script is not an option, then you might see whether shells other than Bash would allow for a finer control over passing the interactive tty to child processes. I personally am no expert on more advanced shells.

If you really must use Bash and really must have this functionality through a function to be run by the interactive shell, then I’m afraid the only way out is to make a helper program in a language that allows you to access tcsetpgrp(3) and sigprocmask(2).

The purpose would be to do in the child (your coproc) what has not been done in the parent (the interactive shell) in order to forcefully grab the tty.

Keep in mind though that this is explicitly considered bad practice.

However, if you diligently don’t use the tty from the parent shell while the child still has it, then there might be no harm done. By “don’t use” I mean don’t echo don’t printf don’t read to/from the tty, and certainly don’t run other programs that might access the tty while the child is still running.

A helper program in Python might be something like this:

#!/usr/bin/python3

import os
import sys
import signal

def main():
    in_fd = sys.stdin.fileno()
    if os.isatty(in_fd):
        oldset = signal.pthread_sigmask(signal.SIG_BLOCK, {signal.SIGTTIN, signal.SIGTTOU})
        os.tcsetpgrp(in_fd, os.getpid())
        signal.pthread_sigmask(signal.SIG_SETMASK, oldset)
    if len(sys.argv) > 1:
        # Note: here I used execvp for ease of testing. In production
        # you might prefer to use execv passing it the command to run
        # with full path produced by the shell's completion
        # facility
        os.execvp(sys.argv[1], sys.argv[1:])

if __name__ == '__main__':
    main()

Its equivalent in C would be only a bit longer.

This helper program would need to be run by your coproc with an exec, like this:

run() {
    exec {in}<&0 {out}>&1 {err}>&2
    { coproc exec grab-tty.py "$@" {side_channel_in}<&0 {side_channel_out}>&1 0<&${in}- 1>&${out}- 2>&${err}- ; } 2>/dev/null
    exec {in}<&- {out}>&- {err}>&-

    # while child running:
    #     status/signal/exchange data with child process

    wait
}

This setup worked for me on Ubuntu 14.04 with Bash 4.3 and Python 3.4 for all your example cases, sourcing the function by my main interactive shell and running run from command prompt.

If you need to run a script from the coproc, it might be necessary to run it with bash -i, otherwise Bash might start with pipes or /dev/null on stdin/stdout/stderr rather than inheriting the tty grabbed by the Python script. Also, whatever you run within the coproc (or below it) would better not invoke additional run()s. (not sure actually, haven’t tested that scenario, but I suppose it would need at least careful encapsulation).

---

In order to answer your specific (sub-)questions I need to introduce a bit of theory.

Every tty has one, and only one, so-called “session”. (Not every session has a tty, though, such as the case for the typical daemon process, but I suppose this is not relevant here).

Basically, every session is a collection of processes, and is identified through an id corresponding to the “session leader”’s pid. The “session leader” is thus one of those processes belonging to that session, and precisely the one that first started that specific session.

All processes (leader and not) of a particular session can access the tty associated to the session they belong to. But here comes the first distinction: only one process at any one given moment can be the so-called “foreground process”, while all the other ones during that time are “background processes”. The “foreground” process has free access to the tty. On the contrary, “background” processes will be interrupted by the kernel should they dare to access their tty. It’s not like that background processes are not allowed at all, it’s rather that they get signaled by the kernel of the fact that “it’s not their turn to speak”.

So, going to your specific questions:

What exactly do "foreground" and "background" mean?

“foreground” means “being legitimately using the tty at that moment”

“background” means simply “not being using the tty at that moment”

Or, in other words, again by quoting your questions:

I want to know what differentiates foreground and background processes

Legitimate access to the tty.

Is it possible to bring a background process to the foreground while the parent continues to run?

In general terms: background processes (parent or not) do continue to run, it’s just that they get (by default) stopped if they try to access their tty. (Note: they can ignore or otherwise handle those specific signals (SIGTTIN and SIGTTOU) but that is usually not the case, therefore the default disposition is to suspend the process)

However: in case of an interactive shell, it’s the shell that so chooses to suspend itself (in a wait(2) or select(2) or whatever blocking system-call it thinks it’s the most appropriate one for that moment) after it hands the tty over to one of its children that was in background.

From this, the precise answer to that specific question of yours is: when using a shell application it depends on whether the shell you’re using gives you a method (builtin command or what) to not stop itself after having issued the fg command. AFAIK Bash doesn’t allow you such choice. I don’t know about other shell applications.

what makes cmd & different from cmd?

On a cmd, Bash spawns a new process belonging to its own session, hands the tty over to it, and puts itself on waiting.

On a cmd &, Bash spawns a new process belonging to its own session.

how to hand over foreground control to a child process

In general terms: you need to use tcsetpgrp(3). Actually, this can be done by either a parent or a child, but recommended practice is to have it done by a parent.

In the specific case of Bash: you issue the fg command, and by doing so, Bash uses tcsetpgrp(3) in favor of that child then puts itself on waiting.

---

From here, one further insight you might find of interest is that, actually, on fairly recent UNIX systems, there is one additional level of hierarchy among the processes of a session: the so-called “process group”.

This is related because what I’ve said so far with regard to the “foreground” concept is actually not limited to “one single process”, it’s rather to be expanded to “one single process-group”.

That is: it so happens that the usual common case for “foreground” is of only one single process having legitimate access to the tty, but the kernel actually allows for a more advanced case where an entire group of processes (still belonging to the same session) have legitimate access to the tty.

It’s not by mistake, in fact, that the function to call in order to hand over tty “foregroundness” is named tcsetpgrp, and not something like (e.g.) tcsetpid.

However, in practical terms, clearly Bash does not take advantage of this more advanced possibility, and on purpose.

You might want to take advantage of it, though. It all depends on your specific application.

Just as a practical example of process grouping, I could have chosen to use a “regain foreground process group” approach in my solution above, in place of the “hand over foreground group” approach.

That is, I could have make that Python script use the os.setpgid() function (which wraps the setpgid(2) system call) in order to reassign the process to the current foreground process-group (likely the shell process itself, but not necessarily so), thus regaining the foreground state that Bash had not handed over.

However that would be quite an indirect way to the final goal and might also have undesirable side-effects due to the fact that there are several other uses of process-groups not related to tty control that might end up involve your coproc then. For instance, UNIX signals in general can be delivered to an entire process group, rather than to a single process.

Finally, why is so different to invoke your own example run() function from a Bash’s command prompt rather than from a script (or as a script) ?

Because run() invoked from a command prompt is executed by Bash’s own process(*), while when invoked from a script it’s executed by a different process(-group) to which the interactive Bash has already happily handed the tty over.

Therefore, from a script, the last final “defense” that Bash puts in place to avoid competing the tty is easily circumvented by the simple well known trick of saving&restoring the stdin/stdout/stderr’s file-descriptors.

(*) or it might possibly spawn a new process belonging to its own same process-group. I actually never investigated what exact approach an interactive Bash uses to run functions but it doesn’t make any difference tty-wise.

HTH

Answer 3

I don't fully understand the question, but here is some of what you are looking for.

Foreground and background are shell concepts.

• A foreground job has access to the tty. It can be suspended by pressing ctrl-z.
• A suspended job can be moved to the foreground (fg), or background (bg).
• A job may be started in the background with «command»&.
• A background job may be brought to the foreground with fg %jobid
• A job is a process, plus other meta-data help by the shell. A job can only be accessed as a job from the shell that started it. From other points of view it is just a process.

Answer 4

It fails, I presume, because Vim detects that it is a background process and doesn't have terminal access, so instead of popping up an edit window it suspends itself. I want to fix it so Vim behaves normally.

It actually has nothing to do with foreground or background.

What vim does is call the isatty() function, which will say that it's not connected to the terminal. The only way of fixing that is to make vim be connected to a terminal. There are two ways of doing that:

• Make sure not to use any redirections for stdout. If you have a redirection, even if you end up redirecting to the terminal in the end, isatty() will point to a pipe and not to a terminal, and vim will background itself.
• Use a pseudo-tty. Yes, I know you said you didn't want that; but if redirections are required, then avoiding pseudo-ttys is impossible.