1

I'd like to copy from /A to /C only paths missing on /B and /C. (Please assume those are paths, and can represent also remote locations e.g. mounted via sshfs)

I wonder... Is there more concise way then writing a loop, iterating over filesystem and making check, file by file ?

Example contents of tree directories /A, /B, /C :

/A:

/A/1abc/qwe
/A/2abc/asd
/A/3abc/zxc
/A/4abc/rty

/B:

/A/2abc/asd
/A/3abc/zxc

/C:

/C/1abc/qwe

expected: to copy from /A to /C only path 4abc/rty:

/A/4abc/rty -> /C/4abc/rty

To illustrate practical examples of /A, /B, /C, leading to such scenario:

  • you make backup, copy from some /source (/A) to some /destination (/B) and some paths failed to be copied. So you want to have copy anyway of missing ones. So you find /another_destination (/C) that can accept them, and you want to copy only missing ones. Here are example limitations why copy from /A to /B could fail: disc got full; limitation of filesystem of /B (like filename lenght), while not present on /C filesystem; etc.
Grzegorz Wierzowiecki
  • 14,264

1 Answers1

1

One solution is to make a union mount of B and C with C as the write branch. Then the problem reduces to copying from A to B+C only files missing on B+C, which can be done with

rsync -a --ignore-existing /A /B+C

For the union mount, one possibility is unionfs-fuse.

unionfs-fuse -o cow /C=RW:/B=RO /B+C

Or unionfs:

mount -t unionfs -o dirs=/C=rw:/B=ro unionfs /B+C
Grzegorz Wierzowiecki
  • 14,264
Gilles 'SO- stop being evil'
  • 829,060
  • By curiosity : why unionfs-fuse instead of in-krenel unionfs ? I guess, it was just hany to write for you but both would do the job - or are there some edge cases where uninfs-fuse would be better for this kind of application ? – Grzegorz Wierzowiecki Jun 21 '16 at 12:49
  • 1
    @GrzegorzWierzowiecki It has just handy for me to write, most union mount systems can handle this. For a discussion of union mount systems, see this thread — when somebody answers! – Gilles 'SO- stop being evil' Jun 21 '16 at 12:56
  • Too many open files (24) by rsync. I wonder if setting higher like -o max_files=32000 option for unionfs-fuse is good strategy? – Grzegorz Wierzowiecki Jun 26 '16 at 21:31
  • No, it doesn't help. Still I get rsync: chown "/.../xyz" failed: Too many open files (24) at some point and rsync does not rsync everything, and unionfs-fuse directory is not possible to ls anymore. Maybe I should try with unionfs or AuFS instead. I couldn't find rsync flag that could help for that. – Grzegorz Wierzowiecki Jun 27 '16 at 14:05
  • @GrzegorzWierzowiecki Oh, I'd never encountered that problem with unionfs-fuse. Presuming that unionfs-fuse is the culprit (the open file limit is per-user), another union mount implementation might help, and an in-kernel one (overlayfs, aufs) would not run into this problem. – Gilles 'SO- stop being evil' Jun 27 '16 at 18:39