Rsync files, excluding a list

Question

I am working on setting up cron jobs,and I am new to this. The purpose is to analyze the new data that is generated.The data is present in directories.

I want to sync only the latest data/that has not been analyzed for which I created a list of directories that have been analyzed(to be excluded).The below command I am using to sync data from source with a certain pattern to destination.

rsync -av --exclude-from=/home/list1 /datasets001/*/Pr*/*RNA* $TMPDIR/raw_fastq

The issue is the above command behaves same as the below command and does not exclude any data mentioned in list1.

rsync -av  /datasets001/*/Pr*/*RNA* $TMPDIR/raw_fastq

few lines of list1 are present below:

datasets001/demux_182_160415_D00163_0376_BHCYVHBCXX_PM155__uid1849/Project_PM155/Sample_PM155_ORG9_1_Case_RNASeq
datasets001/demux_182_160415_D00163_0376_BHCYVHBCXX_PM601__uid1848/Project_PM601/Sample_PM601_PDX10_1_Case_RNASeq

Also,any other suggestions to sync the new data and analyze it(based on date) are welcome.

See http://unix.stackexchange.com/questions/2161/rsync-failing-to-find-files for a guide to rsync filters. — Gilles 'SO- stop being evil', Jun 01 '16 at 22:26

score 2 · Accepted Answer · edited Jun 01 '16 at 22:23

There are two problems;

The first is that you have defined the exclude-from paths as relative paths rather than absolute; try adding a leading '/'.

The second is that you have explicitly provided the source paths, which (I think) is going to override the exclude-from.

If you remove the paths from the exclude list it may work the way you want, ie just have an exclude file like:

Sample_PM155_ORG9_1_Case_RNASeq
Sample_PM601_PDX10_1_Case_RNASeq

Or you could use something like:

rsync -av --exclude-from=/home/list1 \
          --include='/' --include='/*' --include='/*/Pr*' --include='/*/Pr*/*RNA*' \
          --exclude='*' \
          /datasets001/ "$TMPDIR/raw_fastq/"

(note that the order of exclude and include is important, the first match applies)

This will however preserve the source directory structure in the target dir, which may or may not be what you want.

Removing the paths from the exclude list worked.Also,I noticed having more than one newline characters between the lines also works. — Ron, Jun 02 '16 at 14:55

Rsync files, excluding a list

1 Answers1