Bug 1678 - --link-dest doesn't create hard links on remote system.
--link-dest doesn't create hard links on remote system.
Status: CLOSED FIXED
Product: rsync
Classification: Unclassified
Component: core
2.6.0
All Linux
: P3 major
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2004-08-28 18:23 UTC by Richard Neill
Modified: 2005-04-01 11:24 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Neill 2004-08-28 18:23:34 UTC
The following ought to work, and doesn't.

1) On computer "espresso":

rsync -av -e ssh --delete --modify-window=60 --link-dest=../backup.1
home/rjn/bin rjn@pistachio:backup/backup.0

2) On computer "pistachio":

mv backup/backup.0 backup/backup.1

3) repeat (1)


This ought to result in 2 directories on pistachio, each of whose files share
inodes. (i.e. backup.0/bin/foo and backup.1/bin/foo are hard links to each
other). However, they are *not* hard links, but copies!


Some diagnostics:

(i)The above does work  correctly using a different source machine, mocha, (and
in fact several others which I have tried).

(ii)It works OK when the backup is on the same machine (i.e. both on espresso)

(iii)I've tried with very large values of --modify-window, and it makes no
difference. The files in /home/rjn/bin are not being changed at all between
steps 1 and 3

(iv) Rsync versions:
espresso: rsync  version 2.6.3pre1  protocol version 28  (Mandrake 10.1 Beta 2)
mocha: rsync  version 2.6.2  protocol version 28


I apologise if this isn't a bug at all, but results from my own error, however
I've spent several hours testing this to no avail! It does seem to only occur
with machines where the source machine is running a version > 2.6.2. 

Thanks for your help - Richard
Comment 1 Wayne Davison 2004-09-07 12:03:59 UTC
This works fine in my testing.  I'd suggest trying to specify an absolute path
to the --link-dest dir, just in case the relative path is not resolving where
you think it should.  Also, keep in mind that the files in the destination dir
must not already exist for them to be hard-linked with the files in the
--link-dest dir:

If the destination file does not exist, it is first hard-linked with a file of
the same name in the link-dest dir and then the rsync algorithm checks to see if
it needs to be updated.

So, there are two possible failure points, (1) the initial hard link may fail
(if the link-dest file is not found or if the link call fails), or (2) the
hard-linked file is not deemed to be identical with the source file.  One thing
you can do to figure out if the latter is the case is to try leaving out the
preservation of owner and group and see if something about these values are
tripping rsync up.
Comment 2 Richard Neill 2004-09-07 13:59:13 UTC
Thanks for your reply. I've narrowed it down to being caused by -o. Here are the
steps I performed, exactly as they were done, so you can check my diagnosis.
I've used absolute paths as requested.

(I've had to use slightly different machines to do the tests this time:

Rsync versions:
  mocha    rsync  version 2.6.3pre1  protocol version 28
  romance  rsync  version 2.6.0  protocol version 27
  apircam  rsync  version 2.5.7  protocol version 26

---------------------------------------------------------------

[A] This way, (apircam->romance) works as expected:

1)on romance:

   rm -rf /home/rjn/test
   mkdir /home/rjn/test
   
2)on apircam:

   rm -rf /home/rjn/test/source
   mkdir  -p /home/rjn/test/source
   touch  /home/rjn/test/source/file1
   touch  /home/rjn/test/source/file2
 
   rsync -av -e ssh  --link-dest=/home/rjn/test/backup.1 /home/rjn/test/source
rjn@romance.al.cl.cam.ac.uk:test/backup.0


3) on romance:

   ls -i /home/rjn/test/backup.[01]/source/file1
       66 /home/rjn/test/backup.0/source/file1

   mv /home/rjn/test/backup.0  /home/rjn/test/backup.1


4)on apircam:
    
    rsync -av -e ssh  --link-dest=/home/rjn/test/backup.1 /home/rjn/test/source
rjn@romance.al.cl.cam.ac.uk:test/backup.0

5)on romance 

    ls -i /home/rjn/test/backup.[01]/source/file1
       66 /home/rjn/test/backup.0/source/file1  
       66 /home/rjn/test/backup.1/source/file1

6)Success! We have the same inode numbers!

---------------------------------------------------------------

[B] This way, (mocha->romance) fails:

1)on romance:

   rm -rf /home/rjn/test
   mkdir /home/rjn/test
   
2)on mocha:

   rm -rf /home/rjn/test/source
   mkdir -p /home/rjn/test/source
   touch  /home/rjn/test/source/file1
   touch  /home/rjn/test/source/file2
 
   rsync -av -e ssh  --link-dest=/home/rjn/test/backup.1 /home/rjn/test/source
rjn@romance.al.cl.cam.ac.uk:test/backup.0


3)on romance:

    ls -i /home/rjn/test/backup.[01]/source/file1
        66 /home/rjn/test/backup.0/source/file1

    mv /home/rjn/test/backup.0  /home/rjn/test/backup.1


4)on mocha:

   rsync -av -e ssh  --link-dest=/home/rjn/test/backup.1 /home/rjn/test/source
rjn@romance.al.cl.cam.ac.uk:test/backup.0

5)on romance 

   ls -i /home/rjn/test/backup.[01]/source/file1
       70 /home/rjn/test/backup.0/source/file1 
       66 /home/rjn/test/backup.1/source/file1

6)Failure! Inode numbers are different.

---------------------------------------------------------------

Further diagnostics:

1) using -r instead of -a causes [A] to fail as well.

2) using -rlptD instead of -a causes [B] to work.

3) [B] still works if -g is added (i.e. -rlptD)

4) [B] fails with -rlptDo, or -a

5) Here is a stat at the end of step B5:

  File: `/home/rjn/test/backup.0/source/file1'
  Size: 0               Blocks: 0          IO Block: 131072 regular empty file
Device: 2209h/8713d     Inode: 70          Links: 1
Access: (0644/-rw-r--r--)  Uid: (  501/     rjn)   Gid: (  501/     rjn)
Access: 2004-09-07 21:41:22.000000000 +0100
Modify: 2004-09-07 21:13:42.000000000 +0100
Change: 2004-09-07 21:41:22.000000000 +0100

  File: `/home/rjn/test/backup.1/source/file1'
  Size: 0               Blocks: 0          IO Block: 131072 regular empty file
Device: 2209h/8713d     Inode: 66          Links: 1
Access: (0644/-rw-r--r--)  Uid: (  501/     rjn)   Gid: (  501/     rjn)
Access: 2004-09-07 21:41:10.000000000 +0100
Modify: 2004-09-07 21:13:42.000000000 +0100
Change: 2004-09-07 21:41:22.000000000 +0100


6) On all systems, rjn is a regular user, not root. 

7) More details of the computers:

apircam: Mandrake 9.2, kernel 2.4.22-10mdk, ReiserFS
mocha: Mandrake cooker (10.1), kernel 2.6.3-9mdk-p3-smp-64GB, ReiserFS
romance: Mandrake 10, kernel 2.6.3-15mdk-p3-smp-64GB, ReiserFS
  
Comment 3 Wayne Davison 2004-09-17 16:14:04 UTC
Rsync got a bug-fix in 2.6.1 that made --link-dest work properly for a non-root
user when -o was specified and the UID numbers for a username weren't identical.
The receiving rsync is the one that affects this (since it is the one doing the
checking to see if the files are the same and should be linked together or not).

So, in your test that succeeded with -o specified and a 2.6.0 receiving-rsync,
the two systems must have the same UID for that user.

Solutions:  (1) update the receiving rsync to 2.6.3pre1, (2) don't specify/imply
-o, or (3) make sure that your users have matching ID numbers on your various hosts.
Comment 4 Richard Neill 2004-09-17 17:23:30 UTC
Thanks for your explanation. Yes, it's true about the user IDs - they are
accidentally the same on 2 systems, but not the third. 

Can I suggest that rsync might want to emit some sort of warning if the UID
change doesn't work? It might also be worth mentioning the "root only"
requirement of -o in the -a section of the manual. (My mistake, but I didn't
follow the logic of "-a => -o => only root")

Regards

Richard
Comment 5 Wayne Davison 2004-09-17 18:31:47 UTC
I suppose it might be good to note the bug as it relates to older rsyncs, but
there's no problem specifying -o in a modern rsync (rsync just ignores -o if
you're not root).  As I said, this bug with --link-dest and -o was fixed in
2.6.1 (if the receiving side was running something newer than 2.6.0, you
wouldn't have seen this problem).  Things like leaving off -o are just kluges to
avoid the bug if you can't upgrade the receiving rsync.