Portable way to invoke tar with a list of files from stdin

Question

I want to create a .tgz from the content of a directory. I also want to strip the leading "./" from the tar'ed content.
I had done this as follows:

cd /path/to/files/ && find . -type f | cut -c 3- | xargs czf /path/to/tgz/myTgz.tgz

I learned recently that using xargs may not be the best way to pull this off because xargs may invoke tar multiple times if the cmdline arg list gets too long, and I was advised to make use of tar's ability to read a list of input files from stdin. I ended up finding this article on how to do this. However, I find that the recommendation...

cd /path/to/files/ && find . -type f | cut -c 3- | tar czf foo.tgz -T -

...seems to not be portable. It runs fine on my dev PC, but on a busybox target, I get the following error from running the same command:

tar: can't open '-': No such file or directory

So, my question: is there a truly portable/global way to invoke tar to create a .tgz by feeding it input files from stdin (as opposed to cmdline arguments)?

(It is not an option available to me to install alternatives to tar such as gnutar/bsdtar/etc.)

(Secondary question: Why does the "-T -" argument to tar denote "read files from stdin"? From the tar man page, all I could find was that "-T" means:

get names to extract or create from FILE

... but I couldn't see any reference to a plain "-")

Answer 1

It is a common convention to interpret - to mean standard input where an input file name is expected, and to mean standard output where an output file name is expected. Because this is a common convention, the short help summary in the GNU tar man page does not mention it, but the complete manual (usually available locally through info tar) does. The POSIX command line utility syntax guidelines includes this convention, so it's pretty widespread (but it's always a choice on the part of the author of the program).

BusyBox utilities do follow this convention. But the manual does not mention tar as supporting the option -T, and neither does the version on the machine I'm posting this (1.27.2 on Ubuntu). I don't know why you're getting the error “: No such file or directory” rather than “invalid option -- 'T'”. It seems that your tar interprets -T as an option that does not take an argument, then sees - as a file name. Since in this context tar needs a file name to put in the archive, and not just some content that comes from a file, it would not make sense to use the stdin/stdout interpretation for -.

BusyBox utilities support a restricted set of functionality by design, because they're intended for embedded systems where the fancier features of GNU utilities wouldn't fit. Apparently -T is not a feature that the BusyBox designers considered useful.

I don't think BusyBox tar has any way to read file names from stdin. If you need to archive a subset of the files in a directory and you don't need any symbolic links in the archive, a workaround is to create a forest of symbolic links in a temporary directory and archive this temporary directory.

It's not clear exactly why you're using find. If you only want the files in the current directory, why not tar czf /path/to/archive.tgz -- * ? Your command does make sense if there are subdirectories and you want to archive the files in these subdirectories, but not the directories themselves (presumably to restore them in a place where the directory structure must exist but may have different permissions). In this case a leading ./ wouldn't do any harm.

Answer 2

I know that is a stupid answer and maybe not good for you but how bad is if you just...

cd /path/to/files/ && find . -type f | cut -c 3- >/tmp/temp.txt 
tar czf foo.tgz -T /tmp/temp.txt

Answer 3

You can use the -u/--update option in conjunction with xargs. I just tried and to prove that it works, I gave the filename arguments to tar one-by-one using xargs -n 1:

find top_of_tree/ -type f | xargs -n 1 tar -uf archive.tar

Of course, this is just a proof of concept. In your case, it is probably more practical to allow xargs to separate the argument list into larger parts that do not exceed the argument length limit, so as to avoid having a lot of calls to tar:

find top_of_tree/ -type f | xargs tar -uf archive.tar

The prerequisite is that all implementations of tar on your platforms support at least one of the -u/--update flags.

EDIT: the -u/--update flags allow tar to create as well as append to an archive.

I just noticed that unfortunately, you cannot update compressed archives (tgz, created with the -c flag). However, you can, by all means create a non-compressed archive first and then compress the results.

Also, it should not confuse you that I'm using flags to tar with a hyphen ('-' character). It accepts both forms. In my example, -uf would be equivalent to uf in yours.

As for the - option, it is probalby not documented in the man page of tar because it is a very common Unix convention for command line tools that deal with streams as well as named files. - is usually a special argument for filters and names the standard input stream as a file.

Answer 4

This post was helpful, but here's a real-world example for where I only wanted to backup the folder structure in a particular file system, and wanted tar to read said folder structure from stdin:

find /kelinda/ -type d | tar cvf /tmp/kelinda.folders.tar --no-recursion -T -

Note: /kelinda is just a file system that I have which exists on an external hard drive.

So you can use find to find to get a list of exactly what you want and pipe that into tar. The "-T -" will read input files from stdin (as stated in other posts), then I added the "--no-recursion" to prevent tar from backing up other files that I didn't want.

In other words, using find, along with the --no-recursion option, gives you a way to backup an exact list of files and/or folders and then combining that with the "-T -" option allows you to read said list of input from stdin.