Is there a robust way to cache contents of usb on local hd?

Question

I'd like to switch between working on a laptop with a debian vm and a desktop pc running debian while carrying all my work files in a usb. Is there a robust way to automatically start caching the content of the usb (for fast access and search) on the computer I'm using? Both computers have ssds and the extra storage space this (presumably) would require is not a concern, what I want is the speed of ssds and the portability of a usbs without any significant risk of destroying all my files.

score 2 · Accepted Answer · answered Apr 07 '13 at 22:36

2

Depending on the kind of files you are working with, you could use a git repository on the stick.

Just push your work to the stick, move it to the other box and pull all the changes. - Version control and Backup included. :)

answered Apr 07 '13 at 22:36

michas

21,510

... or any other DVCS. :) – peterph Apr 08 '13 at 12:36

score 2 · Answer 2 · answered Apr 08 '13 at 00:14

You can use Unison. From the home page:

Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

Each of the three data storages (laptop disk, desktop disk and USB stick) can be used independently. Before and after use, the USB drive has to be synced. This can be done using the command line tool or one of the available GUIs.

Either you kick off the syncronisation step manually when you mount/unmount the USB disk or you can write an udev rule which does this automatically. On the laptop you sync to a VM shared directory, this allows easy access. Or you can pass the USB stick straight to the guest and sync from within the Debian system.

score 1 · Answer 3 · answered Jul 07 '13 at 20:23

Another options is git-annex. It's designed to do this exact thing, but I've found it clumsy atm from the command line and the webapp. I continue to watch it as it improves.

Git-annex uses git for managing metadata about files and then can use whatever backend you want to avoid storing the actual files in the git repo (which doesn't scale well with file size.)

Checkout these pages:

http://git-annex.branchable.com/ (use case: The Nomad)
http://git-annex.branchable.com/assistant/

score 0 · Answer 4 · edited Apr 13 '17 at 12:36

0

Another option is rsync. But it doesn't really do caching - the data is guaranteed to be identical only when the program finishes (and I do have doubts about the other solutions in this regard - Unison seems to need syncing as well).

You might want to update the question to reflect whether you want caching in the sense, that once you access files in local copy, they are automatically updated before their contents is made available into user-space.

You might be interested in How to auto-sync with a plugged-in USB mass storage device?.

edited Apr 13 '17 at 12:36

Community

1

answered Apr 08 '13 at 12:41

peterph

30,838

thanks for your answer, what you describe in the second paragraph would be nice, do you know of anything that does that? – Apr 08 '13 at 15:03
No, and I'm actually not sure whether something like that can actually be implemented fully in userspace. You may try to search for something inotify-based, though. – peterph Apr 08 '13 at 15:58

Is there a robust way to cache contents of usb on local hd?

4 Answers4