Bug 3175 - it would be nice if --link-dest matched up devices and symlinks too
Summary: it would be nice if --link-dest matched up devices and symlinks too
Alias: None
Product: rsync
Classification: Unclassified
Component: core (show other bugs)
Version: 2.6.7
Hardware: x86 Linux
: P3 enhancement (vote)
Target Milestone: ---
Assignee: Wayne Davison
QA Contact: Rsync QA Contact
Depends on:
Reported: 2005-10-17 04:31 UTC by Lenny Foner
Modified: 2006-03-12 02:56 UTC (History)
1 user (show)

See Also:

Transcript of unsuccessful attempts (2.73 KB, text/plain)
2005-10-17 04:36 UTC, Lenny Foner
no flags Details
Allow --link-dest to affects symlinks and devices (2.59 KB, patch)
2005-10-17 12:27 UTC, Wayne Davison
no flags Details
Updated patch (2.60 KB, patch)
2005-10-17 16:12 UTC, Wayne Davison
no flags Details
Corrected update (2.60 KB, patch)
2005-10-17 16:21 UTC, Wayne Davison
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Lenny Foner 2005-10-17 04:31:56 UTC
[I'm using the version from the 20051014 nightly snapshot, on Ubuntu Breezy
5.10, which is a Debian-based system, and using an ext3fs filesystem.]

It doesn't appear as if rsync correctly hard-links devices together.  This came
up in the context of dirvish runs which happen to include some filesystem trees
of older versions of entire OS disks, so they've got /dev directories in them,
etc.  dirvish is supposed to crosslink snapshots with each other (courtesy of
--link-dest), but it wasn't doing so for these files.  And I can't get rsync to
do it when I invoke it manually, either---it just always creates the devices anew.

I'll either attach (if this bugzilla lets me) or add as a second comment (if it
doesn't) a transcript showing this behavior.  (There are some pointless switches
such as -z in those transcripts because I'm trying to alter the dirvish
invocation of rsync as little as possible in testing.)  This seems like a bug,
since I have no problems manually hardlinking these files together with ln -f,
and it's annoying since it clutters up my logs with thousands of spurious
transfers (even though dev files are small, they shouldn't be getting
redundantly transferred night after night because of this).

Comment 1 Lenny Foner 2005-10-17 04:36:08 UTC
Created attachment 1518 [details]
Transcript of unsuccessful attempts

I have other transcripts with me experimenting with all four possibilities of
trailing or no trailing slash on the link-dest & source directories, just in
case that was confusing things; I couldn't find a combination that worked. 
(And it can't -all- be pilot error, anyway, since dirvish was having the same
problems, and only on the device files.)
Comment 2 Wayne Davison 2005-10-17 12:24:45 UTC
Only regular files are currently hard-linked together via --link-dest.  I have
been hesitant to add to add hard-linking of non-regular files because it will
not always be supported and it doesn't save much space, but perhaps it wouldn't
be too complex to code this up.
Comment 3 Wayne Davison 2005-10-17 12:27:46 UTC
Created attachment 1519 [details]
Allow --link-dest to affects symlinks and devices

I've worked up a patch for testing that appears to handle things pretty well. 
It still needs some improvement, though -- i.e. I need to add some code that
would discern if a failed hard-link attempt means that a symlink or device
cannot be hard-linked, or if just this attempt failed.
Comment 4 Lenny Foner 2005-10-17 15:27:26 UTC
Oh!  I had no idea --list-dest -wasn't- supposed to work with devices (and
symlinks), given that the documentation for the option said nothing about the
restriction and given that rsync does handle devices (and symlinks) correctly in
other contexts.  So I'd say that was a documentation bug... :)

In any event, I've applied your patch (thanks! how speedy! :) and rerun my quick
test---works fine.  (Also passes all the tests in make test, except for two it
rightly skipped [chmod-temp-dir and ssh-basic].)  I'm now rerunning my full-up
dirvish to make sure that works as well; I should know the results in 90 minutes
or so (and will comment again if it doesn't work).

If you're concerned about this not working in all cases (or on all
filesystems---makes it hard to just check at configure time) and maybe tossing
spurious errors if it -doesn't- work, I guess there's always the possibility of
Yet Another Rsync Option to tell it not to try.  I have no particular opinions
on that, except that rsync already has a ton of options and adding more just
makes that problem worse.  I'd certainly say that this should be enabled by
default and disabled only when necessary and with some sort of warning or
option.  (I have no idea if it's safe to hardlink a device if someone -wasn't-
expecting it, but it can only happen as root...  And if you're running with
--link-dest, as root, there's already something fairly unusual going on and I'd
argue that it's unlikely to screw a -running- system since you're making a -new-
hierarchy that certainly isn't being used as a running system at the time...)

Thanks again!
Comment 5 Wayne Davison 2005-10-17 16:12:33 UTC
Created attachment 1521 [details]
Updated patch

There was a bug in the try_link_dest() function when a symlink was being copied
with the same name as a non-symlink in the --link-dest dir.  This version fixes
Comment 6 Wayne Davison 2005-10-17 16:21:20 UTC
Created attachment 1522 [details]
Corrected update

This patch is the right one for the fixed version.  Sorry for the confusion.
Comment 7 Lenny Foner 2005-10-17 17:49:07 UTC
Thanks!  Your first patch worked fine for me in my big dirvish test (because I
didn't have any clashing symlinks---at least, on a very casual inspection), but
I've downloaded the last patch and reapplied.  Still passes make test.  (Haven't
done any other testing, but this version will be used in the next dirvish run,
at least.)  And it's certainly eliminated all the log clutter by not
retransferring all those devices.
Comment 8 Wayne Davison 2005-12-15 16:51:25 UTC
The latest CVS version now has support for hard-linking devices and symlinks via the --link-dest option (and falls back to making duplicates if the OS doesn't allow that).