42

I'm trying to copy the contents of a failing USB thumb drive. If I read the data too fast, the drive's controller chip overheats and the drive vanishes from the system. When that happens, I need to unplug the drive, wait a minute or so for it to cool, plug it back in, and re-start the copy.

I've got an old backup of the contents of the drive, so the obvious way to get the rest of the data is to use rsync to bring the backup up to date, but this runs into the whole "read too fast, the drive vanishes, and I need to start over" issue. Is there a way to tell rsync to only read X megabytes of data per minute? Alternatively, is it possible to tell it to suspend operations when the drive vanishes, and resume when it gets plugged back in?

Mark
  • 4,244
  • 24
    Have you thought about cooling the drive? – AEonAX Dec 28 '16 at 04:46
  • 3
    How full is the drive? If the drive is more than 50% full it is likely better to just mirror the failing drive first and then only start to deal with more complicated stuff on the mirror. – kasperd Dec 28 '16 at 10:19
  • 1
    @AEonAX: While a cool (sic!) out-of-the-box idea, I wouldn't recommend this: you're putting additional thermal stress on the device, and it might fail completely. – Piskvor left the building Dec 29 '16 at 15:32
  • This feels like the geek's equivalent of a first world problem. I have it too, my USB dock gets overloaded by the huge backup job. – Sridhar Sarnobat Mar 05 '21 at 23:36

4 Answers4

49

Unlike DopeGhoti's experience, the --bwlimit flag does limit data transfer, with my rsync (v3.1.2).

test:

$ dd if=/dev/urandom bs=1M count=10 of=data
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0871822 s, 120 MB/s

$ du -h data
10M     data

$ time rsync -q data fast
0.065 seconds

$ time rsync -q --bwlimit=1M data slow
10.004 seconds

(note: my time output looks different to most time invocations (zsh feature), those times weren't edited by me)

Else, perhaps something along the lines of a double -exec in find. I believe that rsync -R should create & copy the parent folders, but if it doesn't, then cp --parents should.

$ find /failing/usb -exec rsync -R {} /somewhere/safe/ \; -exec sleep 1 \;

Note: also check out ddrescue, it might be right what you're looking for :)

apricot boy
  • 1,153
  • 7
  • 11
  • 1
    Yup. DG may have gotten confused by the word "bandwidth" - but that applies to any data-carrying channel, not only network. Man page specifically says: --bwlimit=KBPS limit I/O bandwidth; KBytes per second, no mention of network. Casually browsing the code, this limits the read speed, regardless of destination. Plus, anecdote: I've been using this option for years now to limit local-to-local write speed. https://linux.die.net/man/1/rsync – Piskvor left the building Dec 29 '16 at 15:29
  • 2
    Okay, on second look, it actually counts the number of bytes written and sleeps for a calculated time. io.c, lines 832-3: if (bwlimit_writemax) sleep_for_bwlimit(n); In any case, this would probably work as intended, reading in short bursts and sleeping in-between. I would recommend starting out with a low value and carefully increasing, or just let the slow transfer run overnight. – Piskvor left the building Dec 29 '16 at 15:39
27

A bit of a MacGyver solution but I have had good success with it in the past:
Put an old USB 1 hub between the stick and the computer. No way in hell it is going to copy fast that way :-)

Other possibility: If you've got another stick available put it on another USB port connected to the same root-hub (so it shares bandwidth with your problem stick).
Now start a big write operation to the extra stick so both sticks are competing for bandwidth. For further control you can use ionice to lower the priority on the problem stick even further.

Tonny
  • 1,733
  • 2
    I'm sure there should be a way to make the ports work as USB 1.1 via a software solution, possibly by giving options to the USB kernel module. – André Borie Dec 28 '16 at 18:00
  • @AndréBorie: There is - https://bbs.archlinux.org/viewtopic.php?pid=1342593#p1342593 , but since there are effective throttles in SW, this is a bit of a brute-force solution. – Piskvor left the building Dec 29 '16 at 15:27
  • @Piskvor it negotiates a lower transfer rate with the device, just like using an USB 1 hub would do. I don't see anything "brute force" about it. – André Borie Dec 29 '16 at 18:13
  • Okay, bad choice of word, I admit. But IMHO reconfiguring devices would require special privileges (root or at least %adm), whereas controlling per-process read speed would be simpler. (Plus, there are devices that fail to work outside USB 2.0 at all - in violation of the spec) – Piskvor left the building Dec 29 '16 at 18:17
15

You can use rsync --bwlimit=RATE to throttle the file transfer speed which, based on comments below, actually specifically seems to work by throttling the read speed of the data, which is precisely what you're looking for.

DopeGhoti
  • 76,081
  • 7
    ionice ? https://linux.die.net/man/1/ionice – steve Dec 27 '16 at 23:29
  • 3
    ionice just gives priority, if two things are reading then, by default they get 50/50. ionice can adjust that, but if there is only one thing reading, then it will get 100% . – coteyr Dec 29 '16 at 03:30
  • @coteyr You're right and that is why you need to combine it with a 2nd bandwidth eating process on the same USB root-hub. See my answer. – Tonny Dec 29 '16 at 09:41
  • 3
    Fortunately, your answer is incorrect: --bwlimit sets I/O limit, not necessarily a network limit. Tested many times in various scenarios: this option makes rsync limit itself on read, regardless of the destination. (Found this out in a similar scenario, where fast writes to a slow local device made them fail) – Piskvor left the building Dec 29 '16 at 15:24
  • 1
    @piskvor - thanks for that! That's what I love about this place; I keep learning new things (: – DopeGhoti Dec 29 '16 at 15:28
13

More solutions, in addition to the rsync --bwlimit=100, for 100KiB/s.

  1. Use ddrescue with the --max-read-rate option for a full disk copy. This will also allow you to restart a full disk copy where it left off after an error.

    ddrescue --max-read-rate=100K /failing/usb output.img output.logfile

ddrescue also has lots of other options relevant to data recovery, see the manual here.

  1. Use pv ("pipe viewer") with the --rate-limit option, or the throttle tool to rate limit pipes. pv can also show progress. You can make this work for any tool that uses a pipe or can use a pipe:

    tar -cf - /failing/usb | throttle -K 100 | tar -C /your/outputfolder/ -xvf -

    or 

    tar -cf - /failing/usb | pv --rate-limit 100k | tar -C /your/outputfolder/  -xvf -

    Take a look at the pv manpage for more settings and examples. For throttle, you might want to also try out -w option, which determines how the speed is calculated, see the throttle manpage.

jmiserez
  • 506
  • 4
  • 10