How to trick a command into thinking its output is going to a terminal

Question

Given a command that changes its behaviour when its output is going to a terminal (e.g. produce coloured output), how can that output be redirected in a pipeline while preserving the changed behaviour? There must be a utility for that, which I am not aware of.

Some commands, like grep --color=always, have option flags to force the behaviour, but the question is how to work around programs that rely solely on testing their output file descriptor.

If it matters, my shell is bash on Linux.

Answer 1

A history of toolsets

You are not the first person to want such a tool. People have been wanting such tools for 30 years. And they've existed for almost that long, too.

The earliest tool for this sort of thing was Daniel J. Bernstein's "pty" package, described by Rich Salz as a "Ginsu knife", which Bernstein wrote back at the turn of the 1990s in order to cheat at nethack (sic!). Version 4 of the "pty" package was published in 1992 to comp.sources.unix (volume 25 issues 127 to 135). It's still locatable on the World Wide Web. Paul Vixie described it at the time:

What can I say? It slices, it dices, it washes dishes, it walks the dog. It "just works", meaning that if you follow the directions you'll get a working package without any pulling of hair or gnashing of teeth or other standard porting activities.

Bernstein later updated this, somewhen on or before 1999-04-07, with a "ptyget" package, which he announced:

I've put together a new pseudo-tty allocator, ptyget. An alpha version is at ftp://koobera.math.uic.edu/pub/software/ptyget-0.50.tar.gz. There's a ptyget mailing list; to join, send an empty message to djb-ptyget-requ...@koobera.math.uic.edu. I designed ptyget's interface from scratch. It's much more modular than pty; the basic pty interface has now been split into three pieces:

• ptyget: a tiny, low-level program — the only setuid program in the package — that allocates a new pseudo-tty and passes it to the program of your choice
• ptyspawn: another small program that runs a child process under a pseudo-tty, waiting for it to exit and watching for stops
• ptyio: another, only slightly larger, program that moves data back and forth

The old Ginsu knife pty is now spelled ptybandage, which is a synonym for ptyget ptyio -t ptyspawn; pty -d, for attaching network programs to pseudo-ttys, is now spelled ptyrun, which is a synonym for ptyget ptyio ptyspawn; and nobuf is a synonym for ptyget ptyio -r ptyspawn -23x. I've split off the session management features into a separate package.

That separate package was the "sess" package.

"ptyget" is, incidentally, notable for exemplifying a very early version of, and one of the few published instances of, Berstein's own never-published "redo" build system. dependon is a clear precursor to redo-ifchange.

Usage

ptybandage

ptybandage is what people usually want in a login session. Its primary use case is making programs that are sensitive to whether their standard inputs, outputs, or errors are connected to terminals operate that way even though they are in fact in shell pipelines, or have their standard file descriptors redirected to file.

It takes a command to run (which has to be a proper external command, of course) and runs it in such a way that it thinks that its standard input, output, and error are attached to a terminal, connecting those through to ptybandage's original standard input, output, and error.

It deals with the nuances of running under job control shells, ensuring that a terminal STOP character not only stops ptybandage but also stops the program running attached to the inner terminal.

ptyrun

ptyrun is what people usually want in TCP network servers. Its primary use case is remote execution environments that have not themselves set up terminals, running programs that don't operate as desired when there's no terminal.

It doesn't expect to be running under a job control shell, and if the command being run receives a stop signal it is simply restarted.

Available toolsets

Dru Nelson publishes both "pty" version 4 and "ptyget".

Paul Jarc publishes a fixed version of ptyget, that attempts to deal with the operating-system-specific pseudo-terminal device ioctls in the original that operating systems actually no longer provide.

The nosh source package comes with workalike ptybandange and ptyrun scripts, which use Laurent Bercot's execline tool and the nosh package's own pseudo-terminal management commands. As of nosh version 1.23 these are available pre-packaged in the nosh-terminal-extras package. (Earlier versions only supplied them to people who built from source.)

A few example uses

Jurjgen Oskam using ptybandage on AIX to feed input from a here document to a program that explicity opens and read its controlling terminal for a password prompt:

$ ptybandage dsmadmc <<EOF >uit.txt
joskam
password
query session
query process
quit
EOF

Andy Bradford using ptyrun on OpenBSD under daemontools and ucspi-tcp to make the bgplgsh interactive router control program accessible via the network whilst making it think that it is talking to a terminal:

#!/bin/sh
exec 2>&1
exec envuidgid rviews tcpserver -vDRHl0 0 23 ptyrun /usr/bin/bgplgsh

Further reading

• Daniel J. Bernstein (1992-02-19). v25i127: Generalized interface to pseudo-tty devices. comp.sources.unix. 25 (127).
• Daniel J. Bernstein (1991-10-04). An introduction to session management. Bernstein on TTYs. JdeBP.
• Paul Jarc. ptyget. Paul Jarc's software.
• Jonathan de Boyne Pollard (2016). Daniel J. Bernstein's ptyget toolset. Softwares.
• Dru Nelson. drudru/pty4. GitHub.
• Dru Nelson. drudru/ptyget. GitHub.
• Laurent Bercot. execline. software.
• Jonathan de Boyne Pollard (2016). Daniel J. Bernstein's softwares all in one. Softwares.
• Jonathan de Boyne Pollard (2014). The nosh package. Softwares.
• Jonathan de Boyne Pollard (2012). An introduction to redo. Frequently Given Answers.
• Reyk Floeter. bgplgsh. 8. OpenBSD manual pages.

Answer 2

You might get what you need by using unbuffer.

unbufferis a tcl / expect script. Look at the source if you want. Also note the CAVEATS section in man.

Also note that it does not execute aliases such as:

alias ls='ls --color=auto'

unless one add a trick as noted by Stéphane Chazelas:

If you do a alias unbuffer='unbuffer ' (note the trailing space), then aliases will be expanded after unbuffer.

Answer 3

You can use socat to start your process with a pty connected, and get socat to connect the other end of the pty to a file. Which AFAIU is exactly what you asked:

socat EXEC:"my-command",pty GOPEN:mylog.log

This method will cause isatty called by my-command to return true and a process that relies only on that will be fooled to output control codes. Note that some processes (notably grep) also check the value of the TERM environment variable, so you might need to set that to something reasonable, like "xterm"

Answer 4

There is also a nice solution posted here on Super User by KarlC:

Compile a small shared library:

echo "int isatty(int fd) { return 1; }" | gcc -O2 -fpic -shared -ldl -o isatty.so -xc -

Then tell your command to load this isatty(3) override dynamically:

LD_PRELOAD=./isatty.so mycommand

This might not work for every command out there, may even break some in unexpected ways, but would probably work in most cases.

Answer 5

How about using script(1)?

For example:

script -q -c 'ls -G' out_file

Will save the ls output to out_file with the color codes preserved.

Answer 6

Unsatisfied with the solutions presented here so far, I released the python. She was effective. This solution doesn't require setuid permissions or any actually-insane monkey-patching with shared libraries and LD_LIBRARY_PATH. Save this script somewhere in your PATH. Don't forget to chmod +x it. Suppose you save this script as pty.

#!/usr/bin/env python
from sys import argv
import os
import signal
I've had problems with python's File objects at this low a level, so
we're going to use integers to specify all files in this script.
stdin = 0
stdout = 1
stderr = 2
Include this if passing the command and arguments to fish to
prevent fish from applying any expansions.
#import re
#def fish_escape(args):
def escape_one(arg):
return "'" + re.sub(r"('|\)", r'\\1', arg) + "'"
escaped_args = map(escape_one, args)
return ' '.join(escaped_args)
if len(argv) < 2:
    os.write(stderr,
b"""A tragically beautiful piece of hackery, made to fool programs like ls,
grep, rg, and fd into thinking they're actually connected to a terminal.
Its usage:
pty command [arg1 arg2 ...]
Examples:
pty ls --color -R | less -r
git log -p | pty rg <search terms> | less -r
""")
    exit(255)
We do not use forkpty here because it would block ^Cs from reaching the
child process. And we don't need that.
ptm, pts = os.openpty()
pid = os.fork()
if pid == 0:
    # The child runs this.
    # To get the behaviour we want, we only need to replace the process's
    # stdout with pts. Everything else should remain in place, so that things
    # like ps -eF | pty rg python | less -r will still work as intended.
    os.dup2(pts, stdout)
    # This is not like a subprocess.call(). It replaces the entire child
    # process with argv[1:], meaning execvp will not return! Web search
    # "fork exec" for more.
    os.execvp(argv[1], argv[1:])
    # Use this if calling fish.
    #os.execvp('fish', ['fish', '-c', fish_escape(argv[1:])])
The parent runs this.
If the parent doesn't close the slave end, the script won't be able to
exit. The next read on ptm after the child process terminates would hang
forever because pts would technically still be open.
os.close(pts)
The whole process group gets SIGINT, including the child, so we don't need
to react to it. We'll know when to leave, judging by what the child does.
signal.signal(signal.SIGINT, signal.SIG_IGN)
while True:
    try:
        chunk = os.read(ptm, 4096)
    except OSError:
        break
    try:
        os.write(stdout, chunk)
    except BrokenPipeError:
        # This happens when the parent is piping output to another process in a
        # pipeline, like in pty ls --color -R | less -r, and the receiving
        # process is terminated before the child has exited. If the receiving
        # process is less, this can happen very easily. It happens every time
        # the user decides to quit less before it has displayed all output. So,
        # we need to stop the child process now.
        os.kill(pid, signal.SIGTERM)
        # Also close the child's inputs and outputs, just in case it is
        # blocking on them and can't react to the SIGTERM as a result.
        os.close(ptm)
        break
wait_pid, status = os.waitpid(pid, 0)
exit(status >> 8)

Answer 7

Based on @Amir's answer, here is a script which generates and then includes the library at runtime:

#!/bin/bash
set -euo pipefail

function clean_up {
  trap - EXIT # Restore default handler to avoid recursion
  [[ -e "${isatty_so:-}" ]] && rm "$isatty_so"
}
# shellcheck disable=2154 ## err is referenced but not assigned
trap 'err=$?; clean_up; exit $err' EXIT HUP INT TERM

isatty_so=$(mktemp --tmpdir "$(basename "$0")".XXXXX.isatty.so)
echo "int isatty(int fd) { return 1; }" \
  | gcc -O2 -fpic -shared -ldl -o "$isatty_so" -xc -
# Allow user to SH=/bin/zsh faketty mycommand
"${SH:-$SHELL}" -c 'eval $@' - LD_PRELOAD="$isatty_so" "$@"