Bug 8856 - --hard-links does not handle hard-linked symlinks correctly on FreeBSD
--hard-links does not handle hard-linked symlinks correctly on FreeBSD
Status: ASSIGNED
Product: rsync
Classification: Unclassified
Component: core
3.0.7
All FreeBSD
: P5 normal
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2012-04-11 01:55 UTC by ron (550 5.1.1; Recipient address rejected: User unknown in local recipient table)
Modified: 2016-03-09 09:00 UTC (History)
2 users (show)

See Also:


Attachments
configure.ac patch (832 bytes, patch)
2014-11-17 10:33 UTC, sylvain
no flags Details
syscall.c patch (311 bytes, patch)
2014-11-17 10:34 UTC, sylvain
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description ron (550 5.1.1; Recipient address rejected: User unknown in local recipient table) 2012-04-11 01:55:06 UTC
Creating hard-linked symlinks fails on FreeBSD RELEASE-8.2.

Simple test case:

# uname -sr
FreeBSD 8.2-RELEASE
# ls -liR
total 4
117761 drwxr-xr-x  2 root  wheel  512 Apr 11 10:31 a
117762 drwxr-xr-x  2 root  wheel  512 Apr 11 11:42 b

./a:
total 0
117763 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 f
117763 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 lf
117764 lrwxr-xr-x  2 root  wheel  1 Apr 11 10:30 ls1 -> f
117765 lrwxr-xr-x  2 root  wheel  6 Apr 11 10:30 ls2 -> /x/y/z
117764 lrwxr-xr-x  2 root  wheel  1 Apr 11 10:30 s1 -> f
117765 lrwxr-xr-x  2 root  wheel  6 Apr 11 10:30 s2 -> /x/y/z

./b:
total 0
# rsync -avH a/ b
sending incremental file list
./
lf
s1 -> f
rsync: link "/root/test/b/ls1" => s1 failed: No such file or directory (2)
s2 -> /x/y/z
rsync: link "/root/test/b/ls2" => s2 failed: No such file or directory (2)
f => lf

sent 155 bytes  received 50 bytes  410.00 bytes/sec
total size is 14  speedup is 0.07
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7]
# ls -liR
total 4
117761 drwxr-xr-x  2 root  wheel  512 Apr 11 10:31 a
117762 drwxr-xr-x  2 root  wheel  512 Apr 11 10:31 b

./a:
total 0
117763 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 f
117763 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 lf
117764 lrwxr-xr-x  2 root  wheel  1 Apr 11 10:30 ls1 -> f
117765 lrwxr-xr-x  2 root  wheel  6 Apr 11 10:30 ls2 -> /x/y/z
117764 lrwxr-xr-x  2 root  wheel  1 Apr 11 10:30 s1 -> f
117765 lrwxr-xr-x  2 root  wheel  6 Apr 11 10:30 s2 -> /x/y/z

./b:
total 0
117768 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 f
117768 -rw-r--r--  2 root  wheel  0 Apr 11 10:30 lf
117766 lrwxr-xr-x  1 root  wheel  1 Apr 11 10:30 s1 -> f
117767 lrwxr-xr-x  1 root  wheel  6 Apr 11 10:30 s2 -> /x/y/z
# rsync --version
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 32-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, no iconv, symtimes

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.


So it not only doesn't hard-link the symlinks in the target directory, it does not even create any in the first place.


Following the exact same steps on Debian squeeze for verification:

# uname -sr
Linux 2.6.32-5-686-bigmem
# ls -liR
.:
total 8
193954 drwxr-xr-x 2 root root 4096 Apr 11 10:50 a
193955 drwxr-xr-x 2 root root 4096 Apr 11 11:49 b

./a:
total 0
193956 -rw-r--r-- 2 root root 0 Apr 11 10:49 f
193956 -rw-r--r-- 2 root root 0 Apr 11 10:49 lf
193957 lrwxrwxrwx 2 root root 1 Apr 11 10:49 ls1 -> f
193958 lrwxrwxrwx 2 root root 6 Apr 11 10:49 ls2 -> /x/y/z
193957 lrwxrwxrwx 2 root root 1 Apr 11 10:49 s1 -> f
193958 lrwxrwxrwx 2 root root 6 Apr 11 10:49 s2 -> /x/y/z

./b:
total 0
# rsync -avH a/ b
sending incremental file list
./
lf
s1 -> f
ls1 => s1
s2 -> /x/y/z
ls2 => s2
f => lf

sent 175 bytes  received 70 bytes  490.00 bytes/sec
total size is 14  speedup is 0.06
# ls -liR
.:
total 8
193954 drwxr-xr-x 2 root root 4096 Apr 11 10:50 a
193955 drwxr-xr-x 2 root root 4096 Apr 11 10:50 b

./a:
total 0
193956 -rw-r--r-- 2 root root 0 Apr 11 10:49 f
193956 -rw-r--r-- 2 root root 0 Apr 11 10:49 lf
193957 lrwxrwxrwx 2 root root 1 Apr 11 10:49 ls1 -> f
193958 lrwxrwxrwx 2 root root 6 Apr 11 10:49 ls2 -> /x/y/z
193957 lrwxrwxrwx 2 root root 1 Apr 11 10:49 s1 -> f
193958 lrwxrwxrwx 2 root root 6 Apr 11 10:49 s2 -> /x/y/z

./b:
total 0
193961 -rw-r--r-- 2 root root 0 Apr 11 10:49 f
193961 -rw-r--r-- 2 root root 0 Apr 11 10:49 lf
193959 lrwxrwxrwx 2 root root 1 Apr 11 10:49 ls1 -> f
193960 lrwxrwxrwx 2 root root 6 Apr 11 10:49 ls2 -> /x/y/z
193959 lrwxrwxrwx 2 root root 1 Apr 11 10:49 s1 -> f
193960 lrwxrwxrwx 2 root root 6 Apr 11 10:49 s2 -> /x/y/z
# rsync --version
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.



Apologies if this has already been fixed or/and reported - I did search Bugzilla but could not find anything.
Comment 1 ron (550 5.1.1; Recipient address rejected: User unknown in local recipient table) 2012-04-11 04:05:36 UTC
> What are the FreeBSD and Linux 'a' filesystem types?

FreeBSD: ufs
Linux: ext3

>Please post the list of commands used to create the FreeBSD directory 'a', starting from rm -rf ./a.

# rm -rf ./a
# mkdir a
# cd a
# touch f
# ln f lf
# ln -s f s1
# ln -s /x/y/z s2
# ln -P s1 ls1
# ln -P s2 ls2

I created two symbolic links to check whether the issue has anything to do with the target existing (as with f) or not (as with /x/y/z).
Comment 2 grarpamp 2012-04-11 04:51:57 UTC
Ok, somehow I missed -P, thanks :)

I had meant to exhaustively test all these various
things with -Ha (and optionally --link-dest) back
when I was reporting those sorts of bugs but never
got around to it. Someone really should work up a
make test for it all.

[1] Inode consumers
.
..
dir
file
char
block
fifo
socket
slink

[2] Name entries
hlink (all but dirs are hard linkable)

[3] Meta info
perm
user/group
birth/modify/access/change time

FreeBSD is moving its syscalls towards being able to
change any of the properties in [3] for entities
in [1], there are some left to go. It's not a bad
or nonsensical thing either, just an inode.
Comment 3 ron (550 5.1.1; Recipient address rejected: User unknown in local recipient table) 2012-04-12 00:27:12 UTC
It sure makes sense to handle all those entities in the same way, even though it gets confusing when the same command acts on them in different ways. As in the case of hard-linking symlinks, where the default is to hard-link the symlink target instead of the symlink itself.

Pretty big (t)ask to verify all permutations of configurations, command line options and operating systems, in any case. :-)

Thanks for looking into this, it's much appreciated.
Comment 4 Wayne Davison 2013-05-27 00:18:04 UTC
Look at the test in configure.ac where it checks if link() can hard-link symlinks and see what needs to change for FreeBSD.
Comment 5 awk 2013-08-07 06:55:08 UTC
Same behavior on:

FreeBSD 9.x ZFS (and UFS)
OmniOS (OpenSolaris) ZFS

In fact it often just hangs outright after the "No such file or directory" error message.
Comment 6 Wayne Davison 2013-08-08 16:02:13 UTC
You should note that this bug is awaiting info that nobody has bothered to provide.  Nothing more is going to be done until someone figures out what is needed.
Comment 7 awk 2013-08-08 17:44:36 UTC
Well, under FreeBSD linkat() will hardlink symlinks, i.e. instead of:

link(source, target);

use:

linkat(AT_FDCWD, source, AT_FDCWD, target, 0);

I patched my copy of rsync to use that and it worked when copying files, but not when using --dry-run (still printed errors / hung).

So that's part of it, at least. I plan on digging into it further when I have a chance.
Comment 8 sylvain 2014-11-17 10:32:00 UTC
To expand on the previous post : POSIX previously mandated that link() resolve the target (as FreeBSD does), but some systems (including Linux) did not. So in the last standard linkat() was added (with a new parameter) to disambiguate. So if linkat() is available it should be used, it's available at least on FreeBSD >= 8.0 (released 23 July 2010) and on Linux >= 2.6.18 (released 20 September 2006) and glibc >= 2.4.

Sources: 
http://pubs.opengroup.org/onlinepubs/9699919799/functions/linkat.html
http://manpages.ubuntu.com/manpages/trusty/man2/link.2.html
http://manpages.ubuntu.com/manpages/trusty/man2/linkat.2.html
https://www.freebsd.org/cgi/man.cgi?query=link&sektion=2&manpath=FreeBSD+10.1-RELEASE+and+Ports&arch=default&format=html

I've patched configure.ac and syscall.c of rsync 3.1.1 to use linkat() on my FreeBSD 9.1 system and I haven't have any problems (including with dry run, contrary to the previous post).
Comment 9 sylvain 2014-11-17 10:33:06 UTC
Created attachment 10435 [details]
configure.ac patch
Comment 10 sylvain 2014-11-17 10:34:11 UTC
Created attachment 10436 [details]
syscall.c patch

A #ifdef HAVE_LINKAT should probably be added