POSIX filenames may contain all characters except /, but some filesystems reserve characters like ?<>\\:*|". Using pax, I can copy files while replacing these reserved characters:
$ pax -rw -s '/[?<>\\:*|\"]/_/gp' /source /target
But pax lacks an --delete option like rsync and rsync cannot substitute characters. I'm looking for a simple way to backup my music collection to an external hard drive on a regular basis.
You can make a view of the FAT filesystem with POSIX semantics, including supporting file names with any character other than / or a null byte. POSIXovl is a relatively recent FUSE filesystem for this.
mkdir backup-fat
mount.posixovl -S /media/sdb1 backup-fat
rsync -au /source backup-fat/target
Characters in file names that VFAT doesn't accept are encoded as %(XX) where XX are hexadecimal digits. As of POSIXovl 1.2.20120215, beware that a file name like %(3A) is encoded as itself, and will be decoded as :, so there is a risk of collision if you have file names containing substrings of the form %(XX).
Beware that POSIXovl does not cope with file names that are too long. If the encoded name doesn't fit in 255 characters, the file can't be stored.
POSIXovl stores unix permissions and ownership in files called .pxovl.FILENAME.
I was curious about this so I did a bit of digging.
There are a couple of patches to do this that never made it into rsync, if you fancy compiling from source. Although both are old so they may not still work. This one specifically adds a --fat-filenames option and this one for a --tr option.
These questions also cover something similar - Rsyncing files with special characters to USB FAT32 and https://askubuntu.com/questions/11634/how-can-i-substitute-colons-when-i-rsync-on-a-usb-key. The best bet is probably rdiff-backup which does the substitution, but I can't see any options to have any control over it. There are some other good ideas too though.
A rough idea for a script that would delete and recreate the files each time, if there are not too many, would be something like:
[sudo] updatedb # doing this as root will update the system database
locate -r '^/full/path/to/source/.*[?<>\\:*|\"]' |
sed 's\^/full/path/to/source/\\' >bad_names
rsync --delete --exclude-from=bad_names -av source/ target
cd source
<bad_names pax -rw -s '/[?<>\\:*|\"]/_/gp' target
locate would be faster than find if the database is reasonably up to date.
Finally you could just do this so that the files don't cause any more problems (at least the currently existing ones):
find source/ -regex '[?<>\\:*|\"]' -exec rename 's/'[?<>\\:*|\"]'/_/g' {} +
I would recommend trying that with rename -n first though in case something goes wrong.
--modify-windowoption forrsync. But it does not solve the character substitution. – Frank Jan 11 '14 at 19:56+and possibly;should be in there. Only way to be sure is to test though... – Graeme Jan 11 '14 at 20:25rsyncsays: Invalid argument.stracegave me:open(".foo\\bar.VVfqTX", O_RDWR|O_CREAT|O_EXCL, 0600) = -1 EINVAL (Invalid argument). I guess this is because of the underlying filesystem telling the kernel not to accept the backslash character. – Frank Jan 11 '14 at 21:13