1

I'm not sure what terminology to use here, so bear with me.

I'm writing an application that produces a lot of logs. I thought it would be cool to be able to have my application stream the logs to foo, and allow other processes to read from foo as though it were a stream (like /dev/random).

That is, I'd like to just continuously stream logs somewhere (without having to worry about filesystem stuff like archiving, deleting, etc), and have the ability for other processes to "tap in" to that stream as though it were some endless file.

How do I go about doing this? What is foo?

Ideally:

terminal 1
> mkthing foo
> while :; do echo 'abcdefg' >> foo; sleep 1; done

terminal 2
> echo foo
(outputs "abcdefg" every second)
JS_Riddler
  • 113

1 Answers1

1

The absolutely normal way to look at the logs is to write them to a file. When you want to see the logs, read the file. To watch lines as they get appended to the file, use a command such as tail -f, which, after it reaches the end of the file, keeps it open and watches for extra lines that get appended to it. The option -f is for “follow”, and in less you can have the same effect with F for ”follow“; in other programs this may be known as “tailing” a file because tail is the classic utility that does this. See the tail tag on this site for more information about tailing utilities.

If you absolutely don't want to write the logs to a file, you can run the application in screen or tmux. To run /usr/bin/my_application --some-option in the background, with screen:

screen -S myapp -d -m /usr/bin/my_application --some-option

To watch the logs:

screen -S myapp -d -r

Press Ctrl+A d to detach, i.e. to stop watching the logs. While you're watching the logs, you can also send input to the application, which may or may not desirable. To grant only read-only access, see Is there a way to run 'screen' in read-only mode?

With tmux, to start the application:

tmux new-session -s myapp -d /usr/bin/my_application --some-option

To view the latest logs:

tmux attach -r -t myapp

Press Ctrl+A d to detach. Run tmux attach without -r to be able to interact with the application.

A named pipe is not what you want. I'm just mentioning it because it superficially looks appropriate (and as I write it's popping up in comments). It will do what you describe:

mkfifo foo
while :; do echo 'abcdefg' >> foo; sleep 1; done

in parallel with

cat foo

This is not what you want because there has to be exactly one reader. If there's no reader, the writing side will block. If there are multiple readers, each line (or chunk, depending on how the program produces output) will be seen by exactly one reader, chosen more or less randomly. If there's a reader and it goes away, the writing side will fail to write and will receive a SIGPIPE if it hasn't disabled them.

Gilles 'SO- stop being evil'
  • 829,060
  • Files are the obvious way to go, but I was wondering if there was a way to do it wihout having to manage the files at all (eg: deleting, archiving, etc.) But, seems this is the best option. I had hoped there was some system-level "piping" feature, which would seem pretty useful for having processes "broadcast" data. I guess not. What happens if treat the file as a buffer -- after it reaches a size whenever I add to the end I also remove from the beginning. Would tail -f still work? – JS_Riddler Apr 14 '20 at 19:36
  • @JS_Riddler Yes, it would. There is a system-level piping feature: it's called a pipe. But as I explain, that doesn't work for you, because what you want is not a pipe. A pipe gets clogged if there's nowhere for the content to go out. – Gilles 'SO- stop being evil' Apr 14 '20 at 21:30
  • Having a more robust "pipe" system seems like a missed opportunity. I can image it being very useful for broadcast/subscribe within a single machine. I guess appending to file + cron to cleanup will suffice. – JS_Riddler Apr 15 '20 at 02:17