27

I have recursively transferred many files and folders with scp, using the command:

scp -rp /source/folder myremoteusername@122.10.12.123:/destination/folder

After the transfer is completed, do I need to check whether all files got transferred without any corruption, or does scp take care of it (i.e. displays some error message if any of the file is not correctly transfered)?

Franck Dernoncourt
  • 4,987
  • Unless you get a non-zero exit status from scp and an accompanying error message to stderr, it will have copied everything correctly and completely. – Chris Davies Jan 04 '16 at 21:38
  • a little tl;dr to my post: use rsync if you can. It does copy validations for every file after a transaction has finished, so it's a good idea to use that just to be a little safer. – polemon Jan 04 '16 at 22:35

3 Answers3

33

scp verifies that it copied all the data sent by the other party. The integrity of the transfer is guaranteed by the cryptographic channel protocol. So you don't need to verify the integrity after the transfer. That would be redundant, and very unlikely to catch any hardware error since the data you're comparing against would probably be read from the cache. Verifying data periodically can be useful, but verifying immediately after the transfer is pointless.

You do however need to ensure that scp isn't telling you that something went wrong. There should be an error message, but the reliable indicator is that scp returns a nonzero exit code if something went wrong.

More precisely, you know that the file was transmitted correctly if scp returns 0 (i.e. the success status code). Checking that the exit status is 0 is necessary when you run any command anyway. If scp returns an error status, or if it's killed by a signal, or if it never dies because the system crashes or loses power while it's running, then you have no guarantees. In particular, since scp copies the file directly to its final name, this means that you can end up with a partial file in case of a system crash. The part that was copied is guaranteed to be correct but the file may be truncated.

For better reliability, use rsync instead of scp. Unless instructed otherwise, rsync writes to a temporary file, and moves it into place once it's finished. Thus, if rsync returns a success code, you know the file is present and a correct, complete copy; if rsync hasn't returned an error code then no file will be present (unless there was an older version of the file, in which case that older version won't be modified).

Franck Dernoncourt
  • 4,987
Gilles 'SO- stop being evil'
  • 829,060
3

I've never had a problem with corruption after I scp something but if you're worried about it you can always run md5sum <filename> on both systems to make sure they're the same.

David King
  • 3,147
  • 9
  • 23
3

Per @david-king 's suggestion, this is an md5 based solution for checking the integrity of files after the transfer. Run the following command once after cding to /source/folder on the local machine and once after cding to /destination/folder on the remote host: find . -type f -print0 | xargs -0 -I {} md5sum {} | md5sum. The resulting hash should be identical after a successful transfer.

Update: According to this answer to a similar question on ServerFault, scp does not guarantee file integrity (Please check this answer by @Gilles for details). Alternative to post-transfer checking of file hashes, you can use rsync to transfer files and check its return code.

Update 2: The following only checks if the files and their respective sizes match after the transfer: find . -type f -print0 | xargs -0 -I {} stat --printf="%n %s\n" {} | sort | md5sum

Community
  • 1
Mani M
  • 39
  • 2
    If scp returns 0, then integrity is guaranteed. Furthermore, if scp doesn't return 0, then the problem isn't generic corruption but specifically truncation; verifying the file size is enough (unless there was already an older version of the target file: if scp was interrupted very early then the older version could still be present). Verifying checksums here is a waste of time. – Gilles 'SO- stop being evil' Jan 05 '16 at 01:12
  • Fair. I updated my answer. – Mani M Jan 06 '16 at 03:27
  • You didn't fix the fundamental flaw. Checking scp's return code is enough, and checking hashes is useless. – Gilles 'SO- stop being evil' Jan 06 '16 at 09:05
  • 1
    Even if scp returns 0 it does not guarantee that the file system has not messed things up, so checking the hashes is still useful for checking the integrity of sensitive data if someone choose to use scp over rsync. – Mani M Jan 06 '16 at 18:54