If I try to rsync a file that has an exact size & timestamp match in my
--compare-dest directory, then the file is not created in the output directory.
This behavior is as documented, but it is inconvenient for my application: I
repeatedly rsync a directory of files from a slow server, and I want to use
another directory as a cache on the destination machine without influencing the
correctness of the transfer. I don't want to use --link-dest because it changes
the semantics; it's too easy to mangle a file when it's hard linked. I am
working around the problem by messing up the timestamps of everything in my
compare-dest directory. I think this means that it can use the compare-dest as a
hash matching source, but will still create the output file in all cases.
So this bug is a feature request for a --copy-dest or
--create-the-output-file-anyway-please option. Or maybe it's just my way of
asking if I'm being an idiot.
Yes, it certainly can seem counter intuitive that the files that match in the
compare-dest dir don't get copied into the real destination directory.
A better solution than messing up the timestamps in the compare-dest dir is to
just specify the -I option (which turns off the size+time quick check, causing
all the files to be updated, and the unchanged files will get copied from the
compare-dest dir). It does cause all the matching files to get read on both
machines, though, which could be optimized a little if you are willing to do
For instance, if the amount of changed data is not large, you could first copy
the compare-dest dir to the new destionation and then rsync the data from the
remote system without using compare-dest. This saves the remote read, but is
not optimal if lots of files were changed.
Another potentially workable solution is to start with the same compare-dest
copy that you're doing now, and follow it with an rsync of the compare-dest dir
to the new destination dir using the --update (-u) option. As long as you know
that the changed files will have more recent timestamps than the older files in
the compare-dest dir, all the files in the new destination dir will be left
alone. If that is too risky for you, you could make a note of all the files
that get transferred in the first rsync (i.e. use -v and filter the output into
a file list) and then use that file list as an exclusion list for the extra copy
(input it it into --exclude-from).
I'll also accept this request for an enhancement. If rsync had either the
--copy-dest option you suggest or an --exclude-existing option this would be
easier to do.
The version in CVS now supports --copy-dest (it also supports the specifying of
multiple basis dirs of a single kind, either compare-dest, copy-dest, or link-dest).
I decided to not include --copy-dest in 2.6.4, as I didn't like how my
implementation dealt with the duplication of identical files on the client.
We'll revisit this later.
This option actually made it into 2.6.4 after all (after improvements were made).