1

I currently run rsync --remove-source-files -P. I am transfering a file of about 5TB. This means that significant storage space is wasted at the later parts of the transfer, as the server I am copying from retains the entire 5 TB up until the end. I am looking for a way to prevent that, and make it clear up space during the transfer already. Is this possible?

dockynodyerror
  • 15
  • I like the idea, but it looks like running this would require me to have an additional 5TB of storage on the server. Is there a way around that? – dockynodyerror May 16 '21 at 01:04

1 Answers1

2

I don't know anything within rsync that would facilitate this. Also, the operation in general (free allocated space from the beginning of a file) is not universally supported. Some combinations of operating system and filesystem may allow it. Otherwise, the file can only be truncated from the end. (Probably possible in most linux systems by using fallocate -p)

Some possible options:

  • Split the file. Since you can delete it, it must not be in use on your source machine. Split it into smaller parts and then rsync can delete each individual part when the transfer is complete. If you do this by starting at the end rather than the beginning, you get a wider set of OS that will support it because of general availability of truncate to work on the source file.
  • Manually zero the file. This requires some mechanism to detect how much of the file has been transferred (either out-of-band from the target, or parsing --progress output), then something to call fallocate -p on the copied parts. You'd also want to test that if the copy dies in the middle that you can properly restart it via --partial (probably with --inplace and --append as well)
BowlOfRed
  • 3,772
  • Interesting. I hacked this in the past via truncate, but didn't think of fallocate. – nisc Jul 29 '21 at 17:52