Bug 1479 - request the addition of a --copy-dest option that would copy identical files
Summary: request the addition of a --copy-dest option that would copy identical files
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 2.6.3
Hardware: All Linux
: P5 enhancement (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
Depends on:
Reported: 2004-06-25 17:56 UTC by Steve Fink
Modified: 2005-06-03 01:39 UTC (History)
0 users

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description Steve Fink 2004-06-25 17:56:42 UTC
If I try to rsync a file that has an exact size & timestamp match in my
--compare-dest directory, then the file is not created in the output directory.
This behavior is as documented, but it is inconvenient for my application: I
repeatedly rsync a directory of files from a slow server, and I want to use
another directory as a cache on the destination machine without influencing the
correctness of the transfer. I don't want to use --link-dest because it changes
the semantics; it's too easy to mangle a file when it's hard linked. I am
working around the problem by messing up the timestamps of everything in my
compare-dest directory. I think this means that it can use the compare-dest as a
hash matching source, but will still create the output file in all cases.

So this bug is a feature request for a --copy-dest or
--create-the-output-file-anyway-please option. Or maybe it's just my way of
asking if I'm being an idiot.
Comment 1 Wayne Davison 2004-06-25 19:12:10 UTC
Yes, it certainly can seem counter intuitive that the files that match in the
compare-dest dir don't get copied into the real destination directory.

A better solution than messing up the timestamps in the compare-dest dir is to
just specify the -I option (which turns off the size+time quick check, causing
all the files to be updated, and the unchanged files will get copied from the
compare-dest dir).  It does cause all the matching files to get read on both
machines, though, which could be optimized a little if you are willing to do
some scripting.

For instance, if the amount of changed data is not large, you could first copy
the compare-dest dir to the new destionation and then rsync the data from the
remote system without using compare-dest.  This saves the remote read, but is
not optimal if lots of files were changed.

Another potentially workable solution is to start with the same compare-dest
copy that you're doing now, and follow it with an rsync of the compare-dest dir
to the new destination dir using the --update (-u) option.  As long as you know
that the changed files will have more recent timestamps than the older files in
the compare-dest dir, all the files in the new destination dir will be left
alone.  If that is too risky for you, you could make a note of all the files
that get transferred in the first rsync (i.e. use -v and filter the output into
a file list) and then use that file list as an exclusion list for the extra copy
(input it it into --exclude-from).

I'll also accept this request for an enhancement.  If rsync had either the
--copy-dest option you suggest or an --exclude-existing option this would be
easier to do.
Comment 2 Wayne Davison 2004-11-27 14:27:32 UTC
The version in CVS now supports --copy-dest (it also supports the specifying of
multiple basis dirs of a single kind, either compare-dest, copy-dest, or link-dest).
Comment 3 Wayne Davison 2005-02-25 10:30:04 UTC
I decided to not include --copy-dest in 2.6.4, as I didn't like how my
implementation dealt with the duplication of identical files on the client. 
We'll revisit this later.
Comment 4 Wayne Davison 2005-04-17 23:06:36 UTC
This option actually made it into 2.6.4 after all (after improvements were made).