Bug 12378 - why i cannot exclude dir/files if using option "--files-from"
Summary: why i cannot exclude dir/files if using option "--files-from"
Status: RESOLVED WORKSFORME
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 3.1.2
Hardware: x64 Linux
: P5 normal (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-10-13 15:45 UTC by Alessio
Modified: 2016-10-14 16:57 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alessio 2016-10-13 15:45:14 UTC
I don't understand if this is intended or not, but when I use "--files-from" and  "--exclude"  options together all files are synced anyhow.
eg.:
rsync -av -c --delete --relative --delete-excluded  --exclude=".cache" --exclude="mydirToExclude" /home . 
it correctly doesn't copy the .cache folder.

but:
rsync -av -c --delete --relative --delete-excluded --files-from=myFileList.txt --exclude=".cache" --exclude="mydirToExclude" /home .
does copy everything.

instead using a pattern such as  --exclude="**myDirToExclude**"  
doesn't copy all lines with that pattern

myFileList.txt is something like:
userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/D3BD7BB89AFC5E5C3921C55D38B7DB4F6A6A8C55
userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/996E251B0D179792066F30DEB82476DF9D5E8B15
userName/.cache/mozilla/firefox/e9dwu0jm.default/directoryLinks.json
userName/.cache/winetricks/dotnet40/dotNetFx40_Full_x86_x64.exe
userName/.cache/winetricks/dotnet30/dotnetfx3.exe
userName/.cache/winetricks/dotnet20sp2/NetFx20SP2_x86.exe
userName/.cache/winetricks/msls31/InstMsiW.exe
userName/Downloads/Sentinel_LDK_Run-time_setup/HASPUserSetup.exe
userName/Downloads/Sentinel_LDK_Run-time_setup/readme.html


Thanks,
alessio
Comment 1 Kevin Korb 2016-10-13 16:24:43 UTC
It did not copy the directory you excluded it copied the files within that directory that you explicitly told it to copy and created the appropriate directories to allow that to happen.

IOW, .cache is not relative to userName/.cache/mozilla/firefox/e9dwu0jm.default/cache2/entries/996E251B0D179792066F30DEB82476DF9D5E8B15
Comment 2 Alessio 2016-10-13 22:04:15 UTC
(In reply to Kevin Korb from comment #1)

Excuse me, but i don't get it, what's the difference between excluding (matching) the pattern from a --files-from  and a remote host?

debugging the rsync transfer from the remote host I get:
[sender] hiding directory home/userName/.cache because of pattern .cache

in the other case nothing is being hidden instead  but .cache  is inside my path:
userName/.cache/winetricks/msls31/InstMsiW.exe
Comment 3 Kevin Korb 2016-10-13 22:13:46 UTC
I didn't say anything specific about a remote host.

You excluded .cache then you told it to copy specific files some of which are in .cache.  If there was a file named .cache inside of a directory that you told it to copy then it would be excluded.  If you excluded userName/.cache/winetricks/msls31/InstMsiW.exe then it would be excluded.  Exclude is not patern matched against parts of things you explicitly tell rsync to copy...

kmk@dementia[1%]> cd /tmp
kmk@dementia[2%]> mkdir test
kmk@dementia[3%]> cd !$
cd test
kmk@dementia[4%]> mkdir src dst
kmk@dementia[5%]> cd src
kmk@dementia[6%]> mkdir -p a b c d e cache/c
kmk@dementia[7%]> cd ..
kmk@dementia[8%]> /bin/echo -ne "a\nb\nc\nd\ne\ncache/c\n" > list
kmk@dementia[9%]> cat list 
a
b
c
d
e
cache/c
kmk@dementia[10%]> rsync -vain --files-from=list --exclude=cache src/ dst/
building file list ... done
cd+++++++++ a/
cd+++++++++ b/
cd+++++++++ c/
cd+++++++++ cache/
cd+++++++++ cache/c/
cd+++++++++ d/
cd+++++++++ e/

sent 157 bytes  received 37 bytes  388.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)
kmk@dementia[11%]> rsync -vain --files-from=list --exclude=c src/ dst/
building file list ... done
cd+++++++++ a/
cd+++++++++ b/
cd+++++++++ d/
cd+++++++++ e/

sent 106 bytes  received 28 bytes  268.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)
kmk@dementia[12%]> rsync -vain --files-from=list --exclude=cache/c src/ dst/
building file list ... done
cd+++++++++ a/
cd+++++++++ b/
cd+++++++++ c/
cd+++++++++ d/
cd+++++++++ e/

sent 121 bytes  received 31 bytes  304.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

Cache isn't really being copied it is just being created to make a place for cache/c to be stored when cache/c or c is not excluded.
Comment 4 Wayne Davison 2016-10-13 23:51:07 UTC
Command args (which includes names inside a files-from file) are never excluded by an exclude directive. You told rsync to copy it, so it copies it. Excludes only affect matching of files that rsync finds inside directories that you told it to copy. It is expected that you already trimmed any unwanted files/dirs from your args before you called rsync, since those names are under your control.
Comment 5 Alessio 2016-10-14 09:20:04 UTC
(In reply to Wayne Davison from comment #4)

Well if it's by design I will not go further. But for me it's a missing "feature" because one has to prepare the list beforehand while rsync could do the job "while it's there".  
It's like the impossibility to sync files newer than specific date, which I really miss, but it's another story. Thanks for your help!


(In reply to Kevin Korb from comment #3)

Thanks for the detailed example and explanation it's exactly what happens to me.
And if you remove the "--files-from" you see that "cache" folder is really excluded from the copy.
I was wondering why the behavior is different applying the "--files-from" or not and unfortunately WayneD said it's by design.
Thanks for your help
Comment 6 Kevin Korb 2016-10-14 16:57:33 UTC
If you want to sync files newer than say 3 days ago that is what --files-from is for...
cd /source
find . -mtime -3 -print | rsync -vai --files-from=- . /target

The primary purpose of --files-from is to give rsync the power of find.