0

EDIT — To clarify/summarize, the scenario is the following:

- Context: Large file (1 TB+) on server A, virtually no disk space left on server A, disk utilization on A keeps growing rapidly and that can't be stopped and there's no practical way to add more storage without interrupting production processes

- Goal: Move the "huge file" from A to another machine B, and delete already transferred parts of the file from A's disks while the file is being transferred (the transfer could take a while given the file size, but the disk utilization keeps growing ruthlessly, so we can't just wait for the transfer to finish)

PS: Please note that I'm primarily looking for a mature standard solution, not a bash script / hack. I think it shouldn't be very difficult to come up with something using tools like truncate. However, if there's no standard solution and someone has an elegant bash script (or similar), I'd still be curious to see it.

Original request:

Is there a standard solution to delete big files (think 1 TB+) as they are being transferred via rsync/scp?

The solutions that I've found require extra disk space to first split the file into pieces. However, what if there is virtually no disk space left for these operations?

In the scp/rsync man pages, I only found switches that delete files after they've been fully transferred.

nisc
  • 1,485
  • 1
    The only way to delete the files and have them still transfer is to delete them after they've been transferred which you've already found out how to do. You already have what you need. – Nasir Riley Jul 28 '21 at 01:25
  • 3
    Can you explain the problem you are running into? Or what issue you are trying to solve? Somebody might come up with an alternative solution. – Eduardo Trápani Jul 28 '21 at 02:58
  • 1
    Similar to https://unix.stackexchange.com/questions/649893/rsync-delete-files-during-transfer? – BowlOfRed Jul 28 '21 at 06:26
  • You cannot "delete" a file while it's being transfered: As long as it's not a file on your drive there's nothing to delete. Are you sure you don't just want to ignore the file and not transfer it to begin with? – Marcus Müller Jul 28 '21 at 07:45
  • Do you want to transfer the file on the machine it's copied from, or being copied to? – Wieland Jul 28 '21 at 08:15
  • @EduardoTrápani added summary – nisc Jul 29 '21 at 17:50
  • @BowlOfRed that's the same question, yes! thank you, couldn't find that. – nisc Jul 29 '21 at 17:51
  • Does this answer your question? rsync delete files during transfer – Eduardo Trápani Jul 30 '21 at 01:28
  • @EduardoTrápani well it doesn't answer it, there's no standard solution proposed. I already knew how to hack a bash script to achieve this task, but it feels like a hack. I can't believe there's no standard GNU/Linux tool for this task. – nisc Jul 31 '21 at 20:28
  • 1
    Careful fallocate -p. – Kamil Maciorowski Jul 31 '21 at 20:35
  • @NasirRiley He's talking about deleting portions of the file after those portions have been transferred, thereby reducing the size of the original file while the "scp" is still in progress. The problem that he's trying to solve is one faced by all sysadmins at various points in their career and is a valid issue. – Michael Martinez Aug 13 '22 at 17:11
  • @MichaelMartinez At no point did I ever state that the issue wasn't valid or otherwise make any claims to the contrary of what you said. – Nasir Riley Aug 13 '22 at 18:36
  • @NasirRiley "You already have what you need." ??? No he, doesn't. He precisely doesn't. – Michael Martinez Aug 14 '22 at 20:32
  • @MichaelMartinez That's great. You can send over a cake and balloons if you think it'll help since you know what's necessary. – Nasir Riley Aug 14 '22 at 21:17

0 Answers0