2

I'm in the process of compressing a very large file (several TBs) with the following command, and it's been running for several days.

tar -I pigz -cf /path/to/compressed.tgz /dataset/to/compress

I'm slightly worried that I'll lose these days of progress over the smallest glitch.

For future reference, is there such a compression scheme/command available that allows one to resume where they've left-off? If not, is there an algorithmic reason for not being able to resume the compression where you've left off?

bjd2385
  • 174
  • 1
    I don't know about existent commands, but this is possible. If you are using Huffman coding, one pass would write out symbol weights and form the tree, then a second pass would traverse the input and write the output. So long as you keep track of how far you are, this is trivial to stop and resume. I implemented this compression approach once, but didn't bother to make it resumable since my input files were always tiny – Fox Feb 27 '19 at 05:06
  • @Fox fascinating, I appreciate your input/experience here. I'll look into that – bjd2385 Feb 27 '19 at 06:26

0 Answers0