How to use multi-threading for creating and extracting tar.xz

Question

I use

tar -cJvf resultfile.tar.xz files_to_compress

to create tar.xz and

tar -xzvf resultfile.tar.xz

to extract the archive in current directory. How to use multi threading in both cases? I don't want to install any utilities.

Artem S. Tashkinov · Accepted Answer · 2024-01-22T11:03:00.720

38

tar -c -I 'xz -9 -T0' -f archive.tar.xz [list of files and folders]

This compresses a list of files and directories into an .tar.xz archive. It does so by specifying the arguments to be passed to the xz subprocess, which compresses the tar archive.

This is done using the -I argument to tar, which tells tar what program to use to compress the tar archive, and what arguments to pass to it. The -9 tells xz to use maximum compression. The -T0 tells xz to use as many threads as you have CPUs.

An update from January, 2024:

Even when using the multithreading option, xz barely scales beyond two threads, besides its compression/decompression performance is relatively low.

I highly recommend using ZSTD instead. The command will be:

tar -c -I 'zstd -22 --ultra --long -T0' -f archive.tar.xz [list of files and folders]

Caveats:

This commands needs a lot of RAM, at the very least 5GB. You may want to reduce the compression level, or remove ultra/long options to decrease RAM consumption.
Threading might not be used if you don't have enough source data (less than 1GB). If you still want to use threading for a small amount of data, decrease the compression level from 22 to say 20.

edited Jan 22 '24 at 11:03

answered Sep 06 '20 at 23:25

Artem S. Tashkinov

29,025

1

On my TAR though, -I won't accept arguments, and expects a program only. Maybe you can make a simple shell script that is exec xz -9 -T0 $*? – ZiggyTheHamster Oct 14 '20 at 01:51
This program is used for piping, so $* is redundant. Also, not sure why you want to use exec. – Artem S. Tashkinov Oct 14 '20 at 09:39
it is freaking fast and very less filesize – Thamaraiselvam Oct 28 '20 at 06:53
1

Warning: If you have many CPUs, this will use a lot of RAM..... – starbeamrainbowlabs Nov 24 '20 at 17:54
2

This returns tar: Couldn't open xz -9 -T0: No such file or directory here (macOS). The following from another answer worked: tar -cf - list of files and folders| xz -9 -T0 >| archive.tar.T.xz – aaronk6 Jan 07 '22 at 12:55
@aaronk6 older versions of tar don't support passing arguments to compressors. Probably MacOS has the same limitation. – Artem S. Tashkinov Jan 07 '22 at 13:45

score 29 · Answer 2 · edited Jun 30 '21 at 10:32

You can use XZ_DEFAULTS or XZ_OPT environment variables:

XZ_DEFAULTS is recommended to be used as a system wide configuration, typically set in a shell initialization script.

XZ_OPT is for passing options to xz when run by a script or tool, e.g. GNU tar. See man xz.

Example: using multiple threads (-T0) and max compression level (-9):

XZ_OPT='-T0 -9' tar -cJf resultfile.tar.xz files_to_compress

Reference with a recent GNU tar on bash or derived shell, also see the xz man page

score 4 · Answer 3 · edited Nov 24 '20 at 08:38

4

For older tars, this works:

tar -cf -  list of files and folders| xz -9 -T0 >| archive.tar.T.xz

edited Nov 24 '20 at 08:38

AdminBee

22,803

answered Nov 23 '20 at 15:16

Will Fischer

41
2

How to use multi-threading for creating and extracting tar.xz

3 Answers3

Linked