3

I work in two PCs and I sync all my files from my primary PC to a USB flash memory. In my second PC, I mount the USB flash memory to the same path to work on my files as I were on my primary PC.

Now for the sake of performance and flash memory lifetime, I need to use any type of differencing, aka overlay, aka union, file-system (like unionfs or aufs) to let me use the USB flash disk as read-only and write changes to a temp directory and at the end allows me to write the changes back to the USB flash at once.

Any help? Any hope?

Update:

Thanks for all your answers and comments. I am interested in Linux and my question is: Does any of the above file-systems allow committing the writes to the lower file-systems when required? If yes, how?

Michael Mrozek
  • 93,103
  • 40
  • 240
  • 233
mmonem
  • 384
  • Is using the USB a requirement? You could use dropbox http://www.dropbox.com/ . Also, are doing this so often that writing the the drive so often is actually going to kill it? You could just make backups and wait for it to die and buy another one, USB drives are cheap. – Falmarri Sep 13 '10 at 07:50
  • 2
    I'm sorry what's the question here? you know what you need (want) to do; so what's the problem? is the problem that generally aufs (etc) don't sync back? so you're looking for an FS that does? if so what's the portability requirement? (e.g. does whatever have to work on windows) – xenoterracide Sep 13 '10 at 23:38
  • you might want to say @name so we know we were replied to. – xenoterracide Sep 17 '10 at 04:29

5 Answers5

5

There is a new dm target called "snapshot-merge".

If you format your USB flash memory as a LVM physical volume, and then locate your desired filesystem atop it in a logical volume, you can

  1. Activate a volume group containing your USB flash memory and another LVM physical volume on a local disk.
  2. Create a snapshot of the logical volume on the local disk.
  3. Mount the snapshot, do whatever you want with it, then umount it.
  4. Merge the snapshot back to the origin.

This should achieve close to what you've asked for, although it requires a scratch block device rather than a temporary directory.

Substitute the parts enclosed in {braces} as appropriate.

# Initial setup of the USB drive.
pvcreate /dev/{USB}
vgcreate {removable} /dev/{USB}
lvcreate -n {base} -l 100%PVS {removable} /dev/{USB}
mkfs -t {fs} {...} /dev/mapper/{removable}-{base}

# Initial setup of the scratch device.
pvcreate /dev/{SCRATCH}

# Mounting the device.
vgextend {removable} /dev/{SCRATCH}
lvcreate -s -n {snap} -l 100%ORIGIN /dev/mapper/{removable}-{base} /dev/{SCRATCH}
mount -t {fs} -o {...} /dev/mapper/{removable}-{snap} {MOUNTPOINT}

# Unmounting the device.
umount {MOUNTPOINT}
lvconvert --merge /dev/mapper/{removable}-{snap}
vgreduce {removable} /dev/{SCRATCH}
vgchange -a n {removable}

Untested, but all the LVM commands have manpages so you should be able to figure things out from here. You might need a vgscan invocation in there somewhere, if the volume group doesn't get automatically detected when you plug the USB drive in.

ephemient
  • 15,880
  • Thanks for your answer. I like your approach but could you please show an example (down to Linux commands level). This would be great – mmonem Sep 16 '10 at 17:03
  • @mmonem Pretty straightforward as long as you have the following versions or later: Linux kernel 2.6.33, LVM2 2.02.58. – ephemient Sep 17 '10 at 04:09
  • I liked this approach. I just wonder: is it absolutely necessary to pass the physical volume argument to lvcreate during "Initial setup of the USB drive"? What if I haven't thought that I would add another physical device to the volume group later, and hence didn't care about specifying the physical device explicitly in the call of lvcreate, and then want to use this approach -- how do I proceed before I can do vgextent safely (to be sure that that logical volume will only reside on the removable physical device)? – imz -- Ivan Zakharyaschev Aug 17 '11 at 11:47
2

This seems a use case for dm-userspace+cowd: in essence, you would set up a DM target (block device) consisting of a COW (copy-on-write) file and the block device corresponding to your USB stick, and use it to host a filesystem. All updates would go to the COW file; reads which are not in the COW file would be served off the USB stick; after you unmount the filesystem, merge modifications from the COW file into the USB stick.

Unfortunately, it's Linux specific and development seems to have stopped in 2007.

If what you want to do is sync'ing files across two (or more) PCs, may I suggest that you put your home (or relevant folders) under a versioning system? The usual work cycle becomes like this:

  1. plug in USB stick;
  2. update home directory repository by pulling latest changes from the USB stick;
  3. do your stuff;
  4. commit changes to the versioning system and update repository on the USB stick.

That's only one write to the USB stick. (Although I agree with what others have said, that by the time yoru USB stick wears out, you will probably have bought another -larger- one.)

Community
  • 1
Riccardo Murri
  • 16,468
1

Perhaps, you could make use of the seeding feature of btrfs (pointed out at http://en.wikipedia.org/wiki/Btrfs#cite_ref-16) -- to seed a read-write fs with a read-only base.

Then you'll face the problem of syncing it back to the removable storage.

I don't know of a btrfs-level feature similar to lvconvert --merge; so you could use some higher level tools (like rsync) to merge your changes back, but that would be not so efficient computationally as an fs-level support for this.

Or, there is also the btrfs subvolume find-new command to assist the copying tools in finding things that have changed and need to be sent back.

(Not tested in practice.)

imz -- Ivan Zakharyaschev
  • 15,462
1

I recommend you to use file synchronizing tools. File system level solution to your problem may not be feasible. Check out unison and conduit. As I understand you already have a copy of files in your primary computer. Here is the work flow that i use:

  1. Work and change files on PC_1. After you are done synchronize them to your USB Stick.

  2. Connect the USB stick to your PC_2 and synchronize the content to your PC_2.

  3. Work and change files on PC_2. After you are done synchronize them to your USB Stick.

Synchronizing will be very fast because only the changed files will be rewritten. Also you can write mount and unmount triggers that will automatically make the synchronization.

For a file system solution you can look for some FS with Copy On Write attribute, e.g. btrfs. Taking snapshots and syncing them may be faster and more effective. But I couldn't find any implementation yet. Also working on file system level will not ease your way on conflict resolution.

memin
  • 31
  • 2
  • Thanks for your answer. This is what I am currently do but I am not satisfied with it because I feel my idea is more straight forward. It also requires 4 syncing rounds. Please check the updated question. – mmonem Sep 14 '10 at 16:54
0

I'm assuming this is for a Unix flavor OS, but in case you're interested in a Windows-based solution, I've used Microsoft Mesh, which is a free tool for syncing a variety of files and folders over a series of computers. Another cool feature is the ability to access this "cloud" (their term, not mine) via a web interface. It comes in handy when you're on a remote computer that is not synced, but you would like to access/download certain files.

Jmoney38
  • 546