1

I want to convert all output to the terminal that a certain command generates, using some filter program. How can I achieve this?

The immediate answer might be, “Use a pipeline”. However, too many Unix tools these days deviate from the original Unix approach with its stress on composability by peeking at the context they are running in and then behaving differently depending on whether they output to a terminal or something else and, in the former case, what the capabilities of the terminal are. I want the command to act as if it was outputting to the original terminal, with its capabilities, and just convert its terminal output in addition.

  • 1
    Does script solve your problem? – Panki Jan 03 '24 at 18:43
  • Not something I have tried, but this is exactly what the virtual-terminals do (the terminal windows that you use every day). – ctrl-alt-delor Jan 03 '24 at 18:48
  • If applicable to your situation, putty or mobaxterm can log session, putty can even filter out unprintable output. – Archemar Jan 03 '24 at 19:12
  • The classic tool for this is called expect (a name that's unfortunately hard to search for, but try https://unix.stackexchange.com/search?tab=votes&q=expect%20tcl%20is%3aa&searchOn=3 for examples). Expect is a TCL library, and many languages now have a library, e.g. pyexpect in Python. @Panki script is fine for logging but not convenient to filter the output in real time. – Gilles 'SO- stop being evil' Jan 03 '24 at 19:33
  • As you point out, some programs change their output when it goes to a pipeline or file instead of to a terminal. Therefore there is no universal solution to capturing the output. The solution that works will depend on the behavior of the program you want to capture. Worst case, you may have to select a terminal emulator that can log the session to a file, and then edit/filter the log to something useful. – Sotto Voce Jan 03 '24 at 19:39
  • It seems that some commentors have slightly misunderstood me. Therefore, some clarification: (1) I don’t want to log terminal output. I want to feed terminal output of a command into another command and write the output of the second command to the terminal. The first command must think it is writing to a terminal, though. (2) I want a solution that readily works within shell scripts, not one for Python or whatever other language. (3) I want automatic conversion of output. It’s not about interaction via terminal windows. – Wolfgang Jeltsch Jan 03 '24 at 21:40
  • @SottoVoce, I think that there is a universal solution. It’s not by getting the program to use interactive mode by passing it some program-specific options but rather by making the program use interactive mode by connecting it to a pseudo-terminal. – Wolfgang Jeltsch Jan 03 '24 at 21:42
  • @ChrisDavies, this doesn’t really answer my question. It seems that the only solution I could adapt might be the one involving socat. However, socat is an extremely complicated tool, nothing like a classical Unix tool that does one thing and does it right. It would take quite a bit of effort to understand how exactly make it do what I want. In addition, it has limitations: at least, you can’t have it run arbitrary commands easily, because it can’t cope well with clashes between the command and its own syntax; so it’s difficult to use in a general script. – Wolfgang Jeltsch Jan 03 '24 at 21:46
  • "some clarification" - please fix your question if it's unclear. Updates in the comments are highly likely to get missed – Chris Davies Jan 03 '24 at 21:57
  • 1
    I think that my question was reasonably clear, but certain commentors haven’t read it carefully. For example, I never talked about logging, but about filtering, and filtering should be a concept understood in a StackExchange space dedicated to Unix. – Wolfgang Jeltsch Jan 03 '24 at 22:02
  • 1
    @WolfgangJeltsch sorry, it was not clear from your question that you wanted to feed the filtered output to a second program. Absent an explicit description of the pipeline, people will assume the output is going to the terminal rather than another program. I agree with the comments/answer that point toward /usr/bin/expect as providing the environment to trick the first program into acting as if its output were going to a terminal. Then the expect script or other pipeline commands can filter and read the output as you desire. – Sotto Voce Jan 03 '24 at 23:48
  • Yes, using expect seems to be one solution. Using socat would be another one. My preliminary attempt is socat -u EXEC:⟨command⟩,pty FD:1 | ⟨filter-program⟩. However, if ⟨filter-program⟩ works line-wise, like most Unix text filters (sed, for example), then this doesn’t play well with interactivity. – Wolfgang Jeltsch Jan 04 '24 at 00:45
  • Maybe it’s actually conceptually bad to convert all output of an interactive program. Maybe terminal output should be left alone by any outside tool and specific parts should be manipulated by the respective program. I wanted to convert text that uses LaTeX-like commands to encode special characters into more readable text in output of git diff, git add --patch, etc., but maybe that’s not a good idea, not least because the conversion would affect also parts that are not source code (although LaTeX-like commands would be unlikely to appear there). – Wolfgang Jeltsch Jan 04 '24 at 00:48
  • 1
    Regarding your reply to ChrisDavies, in the question he links there are several answers. Why those questions don't work for you? How don't they answer the question? What's the problem with unbuffer? Or with the isatty trick? If for some reason they don't satisfy your needs, please [edit] your question to provide specific examples of the limitation of those solutions and what would you expect? – aviro Jan 04 '24 at 14:19
  • @WolfgangJeltsch most of your comments are being ignored by people reading the question. If you have updates you need to fix the question so that the updated version will get seen. This isn't a forum – Chris Davies Jan 04 '24 at 14:49

1 Answers1

0

This is how I've been doing it. I'm not sure exactly what you mean by filter, but Command 1 output: of the echo gives you the means to work with the log file.

#!/bin/bash

set -x

my_command() { command1_output=$(command1) echo "Command 1 output: $command1_output"

# Add more commands as needed

}

my_command >> my_logging_file 2>&1

Sometimes set -x is too much info. You can run tail -f my_logging_file from another terminal and watch it in real-time.

Update for comment:
Avoid the log and initiate, piped to tail, to observe immediately.

#!/bin/bash

set -x

my_command() { command1_output=$(command1) echo "Command 1 output: $command1_output"

# Add more commands as needed

}

my_command 2>&1 | tail -f /dev/stdin

Function my_command is executed, and it's stdout and stderr are piped to tail -f /dev/stdin.


Whereas script logs stdout & stderr, it's intended to "record" the entire session and then "played" back. You can use:

script my_session.log
# Your commands here
exit 0

Then, use scriptreplay my_session.log to watch the session.


The expect command is a scripting language for automating interactive programs. (i.e. commands that require user input.)
For example:

#!/usr/bin/expect

spawn my_command expect "Input:" # Replace with expected prompt send "input_value\r" expect eof

This automates interactions with my_command by waiting for the specific prompt, then sending the chosen input.

  • I used the word “filter” in the classical Unix sense: converting standard input to standard output. Typical filter programs would be head, sort, and such. – Wolfgang Jeltsch Jan 03 '24 at 21:47
  • I definitely didn’t mean to write a log. I want to convert terminal output. – Wolfgang Jeltsch Jan 03 '24 at 21:48
  • @WolfgangJeltsch Updated for comment specifics. – JayCravens Jan 03 '24 at 22:06
  • In the distros I've used, /usr/bin/expect has a good amount of overhead from its TCL dependency. Perhaps the python module is better. – Sotto Voce Jan 03 '24 at 23:42
  • @SottoVoce Agreed. More universal, in general, across languages and platforms. I've messed with pexpect a couple of times. – JayCravens Jan 04 '24 at 00:26
  • The problem with your solutions involving pipes is precisely that then the command in question sees that its output does not go to a terminal and thus doesn’t work as if connected to a terminal. However, the desire to have it work as if connected to a terminal is precisely the key point of my question. – Wolfgang Jeltsch Jan 04 '24 at 00:51
  • Tools like expect and socat may be able to solve my problem, because they create pseudo-terminals and have the commands in question interact with them. However, creating pseudo-terminals is not the key point or only point of tthese tools, and as a result they are very complex, actually too complex for a simple task like mine, I’d say. – Wolfgang Jeltsch Jan 04 '24 at 00:54
  • @WolfgangJeltsch I think you have an incorrect assumption on it's simplicity. There's nothing simple about it. I can think of no other way to do it with no /tmp of any kind. It's only possible because tail can use -f /dev/stdin. They key point appears, vaguely, to be continuous output to console and that is the purpose of expect is no interruption. What you're after requires specificity. Without knowing the "certain command", or what it's generating... Answers would've been of greater possibility if there was any actual context or use case given. – JayCravens Jan 04 '24 at 15:03