How do I "cat and follow" a file?

Question

A file is being sequentially downloaded by wget.

If I start unpacking it with cat myfile.tar.bz2 | tar -xj, it may unpack correctly or fail with "Unexpected EOF", depending on what is faster.

How to "cat and follow" a file, i.e. output content of the file to stdout, but don't exit on EOF, instead keep subsribed to that file and continue outputting new portions of the data, exiting only if the file is closed by writer and not re-opened within N seconds.

---

I've created a script cat_and_follow based on @arielCo's answer that also terminates the tail when the file is not being opened for writing anymore.

Answer 1

tail +1f file

I tested it on Ubuntu with the LibreOffice source tarball while wget was downloading it:

tail +1f libreoffice-4.2.5.2.tar.xz | tar -tvJf -

It also works on Solaris 10, RHEL3, AIX 5 and Busybox 1.22.1 in my Android phone (use tail +1 -f file with Busybox).

Answer 2

The problem is that cat is not aware that the file is still being appended. As soon as cat encounters the (current) end of the file it exits.

You have to make wget write to a pipe (or FIFO) in order to avoid this.

wget -O - http://... | tar -xjf -

Answer 3

To read and follow a file from the beginning until interrupted:

tail -fn +1 file

To demonstrate that, try this (assuming Bash with GNU Coreutils):

(while true; do printf . >> /tmp/file; sleep 1; done)&
tail -fn +1 /tmp/file  # (Ctrl-C to interrupt, of course, or otherwise kill it.)
kill %  # Kills the while-loop.

(Note: The +1f mentioned by others is interpreted as a filename, at least in the GNU tail command.)

The above works for a single file. Concatenation of multiple files would not be able to follow all of them deterministically, without hanging on the first. To ‘cat and follow’, following only the last file, one can use process substitution. Here's another demonstration:

printf file1 > /tmp/file1; printf file2 > /tmp/file2
(while true; do printf . | tee -a /tmp/file{1,2} > /dev/null; sleep 1; done)&
cat /tmp/file1 <(tail -fn +1 /tmp/file2)  # (Interrupt or kill it.)
kill %  # Kills the while-loop.

Answer 4

With less +F [My favorite]

Tested on Linux Ubuntu 18.04 with GNU less version 487, as shown by less --version.

You can also use less:

less -N +F path/to/some/growing/log_file.log

You may also want to follow the name of a file instead of its file descriptor by adding the --follow-name option:

# [My favorite command overall]
less -N --follow-name +F path/to/some/growing/log_file.log

This is useful to follow the file named log_file.log, for instance, even when a rotating log system renames this file to log_file.log.1 as it rotates to a new log file to begin logging. WithOUT --follow_name, less would continue to follow the file log_file.log.1, which is now frozen and not growing, whereas WITH --follow-name, less, will see the name changed and automatically open and begin following the new log_file.log file. See here.

The -N shows line numbers. The + causes less to run the command right after the + symbol when it opens. The F command causes it to continually read and load (ie: "follow") the end of the file, which is particularly useful to see growing log files grow. Pressing F (ie: Shift + F) while less is open is identical to pressing Ctrl + End.

To interrupt and stop this continual loading effect while less is running, press Ctrl + C. Now you can go back to being able to use less like normal, scrolling up and down to view data as you desire. Then press q to exit, like normal. Or, you can resume following the file by typing F (Shift + F).

Note: to just open in less and jump to the end of the file but NOT continually load ("follow") the new contents as they are added, use the G command (Shift + G in less if less is already running) instead of F (Shift + F if less is already running):

less -N +G path/to/some/growing/log_file.log

With tail -f [Best option on BusyBox]

Note that if using tail on a regular Linux machine, not an embedded Linux machine with a BusyBox implementation of tail, you can use tail's --follow=name option to do the equivalent of less's --follow-name option described above.

The following was tested with BusyBox v1.31.1, as shown by busybox --help.

The less +F option isn't available on embedded Linux systems running busybox, so for these systems use tail -f instead:

# Just show new contents as they come into the file
tail -f path/to/some/growing/log_file.log
Also print the entire file first, starting at the first line (+1), before
following and loading new contents continually
tail +1 -f path/to/some/growing/log_file.log

It's not quite as convenient as less, but it still works just fine. Press Ctrl + C to kill the output. Then you can scroll up in the terminal to see previous lines in case you need to "pause" the output and view something more-closely. To "resume" viewing the file, press the Up Arrow key of course to recall your previous command so you can easily run the same tail -f command again.

Some other useful options on BusyBox are -s SECONDS and possibly -F. Ex:

# only check and load new contents at the end of the file every 2 seconds
tail -f -s 2 path/to/some/growing/log_file.log

Here is the full help menu:

# tail --help
BusyBox v1.31.1 (2021-11-20 02:33:23 UTC) multi-call binary.
Usage: tail [OPTIONS] [FILE]...
Print last 10 lines of each FILE (or stdin) to stdout.
With more than one FILE, precede each with a filename header.
-f      Print data as file grows
-c [+]N[kbm]    Print last N bytes
-n N[kbm]   Print last N lines
-n +N[kbm]  Start on Nth line and print the rest
-q      Never print headers
-s SECONDS  Wait SECONDS between reads with -f
-v      Always print headers
-F      Same as -f, but keep retrying


N may be suffixed by k (x1024), b (x512), or m (x1024^2).

With watch [also works on BusyBox]

Here is another option. This also works fine in BusyBox:

# Continually view the last 20 messages of the log file every 1 second
watch -n 1 'tail -n 20 path/to/some/growing/log_file.log'

References

1. I first learned about less +F here: Open `less` scrolled to the end
2. I first learned about less +G here: Open `less` scrolled to the end
3. Where I learned about less's --follow-name option: https://unix.stackexchange.com/a/196349/114401