1

I'm trying to copy a 100GB zip file via scp, but it keeps disconnecting due to the large file size.

What is the best solution to overcome this?

Artem S. Tashkinov
  • 29,025
Selva Praveen
  • 11
  • 2
    does it perhaps always fail after 4GB? – Jaromanda X Jul 05 '23 at 09:49
  • 2
    What's the exact error message that you're seeing/getting? – Artem S. Tashkinov Jul 05 '23 at 12:40
  • 4
    "What is the best solution to overcome this" - fix the network issue that's causing the connection to break – Chris Davies Jul 05 '23 at 12:45
  • You could try lftp which supports sftp (scp also uses sftp these days) and automatic reconnect and resume upon loss of connection. – Stéphane Chazelas Jul 05 '23 at 13:16
  • i've done 700+gb tar files. scp does not drop on its own, and it is nnothing in /etc/ssh/sshd_config that would be the problem the default settings would be just fine. It is a network problem like was said. In my case, this one time (at band camp) there was a duplicate IP address on the network not easily found that was causing scp disconnect to happen on me after about 2 minutes into scp transfer. – ron Jul 05 '23 at 14:01
  • it would of course disconnect if your 100gb zip file exceeds the destination storage capacity or has some kind of disk quota on the destination (i.e. home) folder. – ron Jul 05 '23 at 14:03

1 Answers1

3

Here are some options.

1. Use the -Cflag:

The -C flag enables compression during the file transfer.

scp -C /source/file.zip user@host:/destination/file.zip

2. Increase the SSH timeout on the server:

You can modify the ClientAliveInterval and ClientAliveCountMax options in the SSH server configuration file /etc/ssh/sshd_config on most systems.

3. Split the file into smaller parts:

Another approach is to split the large file into smaller parts using a file archiving tool like split or 7z.

4. Use rsync:

Rsync is a powerful tool for file synchronization and transfers. It has built-in resumable transfers and can efficiently transfer only the parts of the file that have changed. You can use rsync instead of SCP to copy the file and resume the transfer in case of interruptions.

5. SFTP:

SFTP is an extension of SSH that provides secure file transfer capabilities. It can be more reliable than SCP for large file transfers.

6. SSHFS/FUSE | This is my recommendation:

I would use SSHFS/FUSE. With this, you can mount the directory from the target server directly into a directory on your own system, allowing you to work as if the target folder is present, local on the client.

Z0OM
  • 3,149