0

I have large directory that contains several millions+ sub-dir, each sub-dir contain 3 or less image files. What is the most efficient way for me to zip them so I can move it to a new server?

The sub-dir are 5 letter words compose of random letters + numbers.

For ex:
/foo/re63d/image.jpg, /foo/re63d/image2.jpg
/foo/4sf3g/image.jpg, /foo/4sf3g/image2.jpg
/foo/vj33s/image.jpg, /foo/vj33s/image2.jpg
and so on several millions times

I was thinking of using shell script to zip/tar them alphabetically. For ex every folders starting with

00* zip them into 0_0.zip
01* zip them into 0_1.zip
a0* into a_0.zip
a1* into a_1.zip
aa* into a_a.zip

This way it's easier for me to manage them. I want to be able to choose what sub section I want to zip at a time like aa* to af* as I want to be able control how long each process run. How would I go about doing this?

Rui F Ribeiro
  • 56,709
  • 26
  • 150
  • 232
Perz
  • 11
  • 4
    Not a real answer, but if what you want is to reliably move a lot of files to another server, I'd say rsync would be a better tool. – nxnev Dec 21 '18 at 23:34
  • I like the ability to see what I transfer. How much should I be worry about the integrity of the transfers if there's a broken connection? – Perz Dec 21 '18 at 23:43
  • 1
    @nxnev I would say it constitutes an answer because it's better for OPs use-case (It's not what he/she wants, but what he/she needs.) Perz: rsync is rock solid and will continue where it left off when there is a broken connection and has a compression algorithm built-in – Fabby Dec 21 '18 at 23:48
  • Thank you. I will have a look at rsync then. – Perz Dec 21 '18 at 23:52
  • What do you mean by efficient: Smallest size, least cpu time, least effort my you? – ctrl-alt-delor Dec 22 '18 at 12:45
  • Most efficient by fastest way of doing it without sacrificing data integrity? – Perz Dec 22 '18 at 15:06
  • I found this command tar -cz /startdir | pv | ssh user@ip "cd /destpath; tar -xf" that seem to fit my need, my question is is it worth it to compress such huge number of directories and files? won't it be faster if I just remove the z compression? – Perz Dec 22 '18 at 15:24
  • compressing is a waste of time for images, as they are already compressed. – matzeri Dec 24 '18 at 13:41

0 Answers0