Copy large volume of files whilst preserving folder structure

Question

I'm copying about 300K files (about 30MB for each file) from one directory to another. I used the following command to preserve the folder structure

find . -name '*.msg' -exec cp -p --parents \{\} /destinated_folder \;

I'm facing this error message

packet_write_wait: Connection to aa.bbb.ccc.dd port ee: Broken pipe`

The script is running but somehow it stopped halfway. I have also seen similar discussion using rsync in here.

I wonder what might be the solution? I'm using Ubuntu with 6vCPUs and 32GB memory.

Have you looked further into rsync? It seems to do a good job picking up from partial-runs. — DopeGhoti, Jul 01 '20 at 18:35

score 0 · Accepted Answer · answered Jul 01 '20 at 18:53

If you use find -exec cp you will have a process that cannot easily be restarted in the vent of an error. I'm not convinced that the error you're seeing it's anything to do with the command you've shown us, as there's no networking referenced in the command.

Maybe you're copying from or to an SMB/NFS drive? (If so, you'll get a more efficient copy if you can log on to the remote system and let rsync manage the network copying.)

Nevertheless, this rsync command will allow you to restart the copy process multiple times (including handling the copies you've already made)

rsync -av --include '*.msg' --include '*/' --exclude '*' --prune-empty-dirs --dry-run . /destinated_folder/

Remove --dry-run when you are happy that it's going to do what you expect.

thanks and yes I was copying files from the SMB drive to the Ubuntu machine. The process was taking hours. I'm guessing the connection breaks in between and hence I saw this error message. — byc, Jul 02 '20 at 13:12

Copy large volume of files whilst preserving folder structure

1 Answers1