rsync : Copy symlinks as regular files

Question

I need to copy a large directory containing all kinds of files onto a drive that already has a lot of important data.

I'd like to be able to copy munged symlinks but the drive is exfat and symlinks are not supported. Because the file already has a lot of data on it, I'd like to avoid having to transfer that data in order to format it to something that does support symlinks.

Is there a way that rsync could read a symlink such as this:

/etc/apache2/sites-enabled/mysite.conf -> ../sites-available/mysite.conf

and generate a regular file like

<rsync-destination>/etc/apache2/sites-enabled/mysite.conf.rsync-munged

containing the text

../sites-available/mysite.conf

I would prefer if the solution was something built into rsync or done in bash pipeline, or at least part of the basic tools found on any linux, but I'm willing to accept an other software that offers what rsync does + this functionnality

Edit:

I see that rsync can not do such a thing. I think I found a way to do it manually:

rsync everything but the symlinks, then seperately run a 'find' operation to identify all symlinks in the source, and manually create them with a bash script.

I'd have to manually extract the symlink data from stdio and I'm not sure how to do that. ls -l has a lot more data that would get in the way

Would it be an option to create an archive (e.g. tar or zip or other), and then transfer that instead? — Kusalananda, Nov 11 '21 at 18:56
that's a good idea, but I need to have it so I can relatively easily read through it and analyze it as a folder structure. I'll look into how easy it is to explore tar archives for my use case. The space on the server is limited as well, so I can not create the file there (note that I'd like to only update and download newer files). If rsync has a direct way to pipe to tar without going through the local filesystem that would be sweet. — sybog64, Nov 11 '21 at 19:02
Well, you don't really need rsync, you could just tar -c -f - directory | ssh remote 'cat >directory.tar' or the other way around, ssh remote 'tar -c -f - directory' >directory.tar. — Kusalananda, Nov 11 '21 at 21:07
No. This will copy everything regardless of changes. the directory is massive. this will not work for me — sybog64, Nov 11 '21 at 22:08

score 1 · Answer 1 · answered Nov 11 '21 at 18:15

What you want isn't directly possible:

You can't convert a symlink into a file with a suffix name such as munged
You can convert symlinks into regular files (--fake-super) but the metadata associated with them that enables them to be converted back into symlinks cannot be stored on an exFAT filesystem

Other possibilities

The --munge-links option leaves symlinks as symlinks but fixes up the value of the symlink so that it cannot be valid on the target system. Not useful for an exFAT filesystem
You can use --copy-links to expand a symlink to the actual file. This probably isn't what you want as the symlink information is lost and cannot be reconstructed when data is returned to the source system

The only other solution I can see is for you to create a large file on the target system that can be mounted as a native Linux filesystem such as ext4. You can then send your backups to that filesystem and benefit from its extra features over and above the limited set in exFAT.

sybog64 · Accepted Answer · 2021-11-14T12:25:16.470

rsync regular files, then seperately find all links, list them in a file and create all the link files one by one

#!/bin/bash
$SRCDIR="/backup/";
$SRCSERVER="user@example.com";
$DEST=".";
rsync -rtpEv "$SRCSERVER:$SRCDEST" $DEST;
ssh SRCSERVER "stat -c %N \`find $SRCDIR -type l\` 2> /dev/null" > links;
$MUNGED = ".munged";
while read p; do

    file=$(echo $p | awk -F ' ' '{print $1}' | sed "s/^`/;s/'$//");

    dest=$(echo $p | awk -F ' ' '{print $3}' | sed "s/^`//;s/'$//");

    echo $dest > "$file$MUNGED";
done < links;

(unsure with my bash syntax, I essentially ran all this manually)

score 0 · Answer 3 · answered Nov 11 '21 at 19:18

If the files don't need to be readable or modifiable directly, just restored as a whole, a simple solution is to archive them with tar or the like.

If the files don't need to be directly readable from the backup, but you want to be able to update the backup, and you have some spare space on the disk, you can use Git to archive the data. As a bonus, you'll be able to recover old versions.

cd /path/to/back/up
git init .
git commit -a
git clone --bare .git /media/external-disk/my-site.git

Git stores symbolic links, but not permissions. If the permissions of the files matter, you can use etckeeper. It wraps around Git (or other version control system) and stores file metadata in a separate file.

If the files don't need to be accessible from a non-Linux system, and you have a good estimate of their total size, you can create an ext4 filesystem image and loop-mount it. Something like this for the initial preparation:

truncate -s 42G /media/external-disk/linux.fs
mkfs.ext4 -F /media/external-disk/linux.fs

Then to use the filesystem image:

sudo mkdir /media/external-linux
sudo mount -o loop /media/external-disk/linux.fs /media/external-linux
sudo rsync -ax / /media/external-linux/my-site-backup
sudo umount /media/external-linux

A potential approach would be a filesystem layer that uses a separate file to store metadata that the underlying filesystem doesn't support. [Upfs](You could try Upfs, does that, but unfortunately, it's unmaintained.

(A long time ago, umsdos was popular; it allowed storing Linux files on a FAT filesystem which didn't support symlinks, permissions, file names longer than 8 characters plus a 3-character extension, etc. Unfortunately it isn't compatible with FAT32 or exFat.)

rsync : Copy symlinks as regular files

3 Answers3