Transferring 70TB data from one remote server to another

Question

First I will elaborate what is the scenario here.

We have 2 servers both are ubuntu 14.04 LTS and we have a drive called /storage/ of 70TB It includes many files of 30GB size each and other ones as well. So as both are the remote servers and I want to move all this data to my other remote server's same drive as /storage/.

Is there any way to do it fastly and stably so that there will be no data loss in that?

once i have tried to move only one file from one to other which worked fine with this link. Any help will be thankful.

Thanks for -1 but am talking about 1: moving NOT copying. 2: It was smal file of few kps now am talking about 70TB — Junaid Farooq, Nov 30 '16 at 12:13
@ijunaidfarooq Moving in this respect is a two step procedure of 1) copying 2) removing the original data. Use scp or rsync - double checking the m5d checksum is not a necessity (see here). Just make sure scp finished correctly and then you can go on deleting the original files. — FelixJN, Nov 30 '16 at 12:19
Don't you what is the difference between copying and moving? Dont you know is meant by 34 byte and 70TB? — Junaid Farooq, Nov 30 '16 at 12:20
Are the two servers in the same physical location? If not, seriously consider carrying the data on a tape (or several) rather than migrating it over a network. You really need to do the maths to see how long it will take to copy that much data by the available network bandwidth before you bother to take that route. Please can you update your question to give some physical details (same datacentre? available bandwidth? single 70TB file or big filesystem? available downtime on source to ensure an accurate copy?) I move tens of TB fairly regularly so I don't think I'm way out of your league. — Chris Davies, Nov 30 '16 at 13:20
https://what-if.xkcd.com/31/ and... perhaps you should consider a really wide bandwidth with a very slow connection speed. Then just delete it off the old location. — Jeutnarg, Apr 28 '17 at 16:06

score 4 · Accepted Answer · answered Nov 30 '16 at 12:49

Is there any way to do it fastly

It depends of the network connection speed between source and destination server.

70 TB is a lot of data. It might be worthy physically disconnecting the drive from the server and remounting it on the destination server.

and stably so that there will be no data loss in that?

If you copy the files via scp their integrity is ensured by the crypto protocols used by the program itself. So as long scp finishes with a zero status, you know that everything went well.

score 0 · Answer 2 · answered Nov 30 '16 at 15:51

I'm not sure which of solutions have you used from mentioned question but I would strongly suggest using rsync and screen as this operation will take a lot of time.

With this amount of data something might go wrong (for example, connection might be lost) and you will take hard time finding out the point where you should restart your procedure. Rsync does it for you.

Talking about speed - well that depends on network connection and, yes, moving physical drive might be also a solution. Take a look at compression switch in rsync it might help if your files can be compressed easly. If not, then going without compression also might be a good idea (no unnecessary CPU usage).

rsync seems the more sensible idea. – Rui F Ribeiro Nov 30 '16 at 15:57 — Rui F Ribeiro, Nov 30 '16 at 15:57

score 0 · Answer 3 · edited Nov 30 '16 at 16:13

0

Simply

Make a backup
Configure the new server, ssh access as root (temporary and with public key only) and install rsync, create dir /storage
transfer using rsync (on slackware ssh is enabled default, if not enable use rsync -e)
After transfer check with df, du if is all ok and disable ssh access as root (mantein only ssh as user)

edited Nov 30 '16 at 16:13

Chris Davies

116,213
16
160
287

answered Nov 30 '16 at 15:53

elbarna

12,695

rsync is fine for smalier amounts of data but will take forever on 70 Tb of data. – fpmurphy Nov 30 '16 at 17:02
2

@fpmurphy1 it depends on so many factors that the question asker hasn't provided. I moved several TB of data from New Zealand to France earlier this year. It took a month (so would have been faster with a disk posted through the mail) but time wasn't important. – Chris Davies Nov 30 '16 at 17:11

Transferring 70TB data from one remote server to another

3 Answers3