Bug 7670 - rsync --hard-links fails where ditto succeeds
rsync --hard-links fails where ditto succeeds
Status: NEW
Product: rsync
Classification: Unclassified
Component: core
3.1.0
Other Mac OS X
: P3 major
: ---
Assigned To: Wayne Davison
Rsync QA Contact
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-09-09 19:33 UTC by Dave Yost
Modified: 2011-02-21 04:54 UTC (History)
1 user (show)

See Also:


Attachments
test script that demonstrates the bug (5.32 KB, text/plain)
2010-09-09 19:40 UTC, Dave Yost
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dave Yost 2010-09-09 19:33:14 UTC
Given enough hard links in a source folder and a small enough destination volume, copying with rsync -- hard-links can fail whereas the Mac OS X ditto program succeeds. Rsync fails by exhausting the maximum number of hard links on the destination.

I have made a script that demonstrates this bug. It makes two minimal ram disks, with a volume on each. On the source volume it makes a maximal set of hard linked files that can be copied correctly to the other volume using ditto but not using rsync.
Comment 1 Dave Yost 2010-09-09 19:40:59 UTC
Created attachment 5956 [details]
test script that demonstrates the bug
Comment 2 Dave Yost 2010-09-09 19:42:31 UTC
test script output looks like this

rsync: mkstemp "/Volumes/Copy/links/03/29/.83b.QlmOQ8" failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.84b.KFW6da" failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.85b.vB6pC5" failed: No space left on device (28)
rsync: link "/Volumes/Copy/links/03/28/36a" => links/03/28/36b failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.86b.jBXuVJ" failed: No space left on device (28)
rsync: link "/Volumes/Copy/links/03/28/37a" => links/03/28/37b failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.87b.C3svVR" failed: No space left on device (28)
rsync: link "/Volumes/Copy/links/03/28/38a" => links/03/28/38b failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.88b.RcRxZH" failed: No space left on device (28)
rsync: link "/Volumes/Copy/links/03/28/39a" => links/03/28/39b failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.89b.mFP87z" failed: No space left on device (28)
rsync: link "/Volumes/Copy/links/03/28/40a" => links/03/28/40b failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.90b.GQbvqB" failed: No space left on device (28)
rsync: link "/Volumes/Copy/links/03/28/41a" => links/03/28/41b failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.91b.L5bJ8n" failed: No space left on device (28)
rsync: link "/Volumes/Copy/links/03/28/42a" => links/03/28/42b failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.92b.PXp0X0" failed: No space left on device (28)
rsync: link "/Volumes/Copy/links/03/28/43a" => links/03/28/43b failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.93b.vpOJOO" failed: No space left on device (28)
rsync: link "/Volumes/Copy/links/03/28/44a" => links/03/28/44b failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.94b.8Y8sFs" failed: No space left on device (28)
rsync: link "/Volumes/Copy/links/03/28/45a" => links/03/28/45b failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.95b.41bN0l" failed: No space left on device (28)
rsync: mkstemp "/Volumes/Copy/links/03/29/.96b.rMQnZB" failed: No space left on device (28)
...
Comment 3 Andrew Gideon 2010-09-14 08:33:41 UTC
Is this really an rsync bug?  I experienced the same thing on Linux with ext3 with our backup system, which is heavily dependent upon --link-dest.  I've tested switching to xfs, and this solves the problem because the real issue isn't rsync but instead the underlying file system.

I also confirmed that cp -Rl had the same problem and solution.

http://www.mail-archive.com/rsync@lists.samba.org/msg25950.html

I did wonder whether rsync should, when faced with this error, fall back on creating a copy.  But should rsync include behavior that exists only to work around a file system limit?  Perhaps only as a command line option (ie. definitely not the default behavior)?

I'm also not sure that it should always do this.  It's not so bad to do so for --link-dest, which is really just an optimization.  But if rsync is to be making a reliable copy of the source at the destination, this should include mimicry of any links.  In that case, falling back to copy would be a bad idea, I believe.

Comment 4 Dave Yost 2010-09-14 12:47:00 UTC
On Mac OS X, the ditto program works where rsync fails, so it seems this is a problem with rsync (and now we know it is also a problem with "cp -Rl"). Wayne, perhaps you could ask a friend at Apple how ditto does the equivalent of --hard-links.
Comment 5 Dave Yost 2011-01-27 18:37:03 UTC
Why is this not a blocker? Why is it still marked New?
Comment 6 Matt McCutchen 2011-01-27 18:48:14 UTC
(In reply to comment #5)
> Why is this not a blocker?

It does not "block development and/or testing work" (https://bugzilla.samba.org/page.cgi?id=fields.html#bug_severity).  An example of a blocker would be if rsync were unable to add directories to the file list at all; then very little meaningful testing could be done.

> Why is it still marked New?

No one has started to work on it.
Comment 7 Jon Bendtsen 2011-02-18 06:38:21 UTC
I experience the same problem on Linux rsync'ing over the network. Both hosts uses XFS filesystem(s), both uses Volumegroups that has been lvextended and then xfs_growfs.

Source uses several smaller LV in different VG (some SSD, some HDD), all XFS and mounted in various locations.

Destination uses 1 big LV in 1 VG.

rsync source: tried both debian lenny and squeeze packages, 3.0.3-2 and 3.0.7-2.
rsync destination debian lenny package, 3.0.3-2

Kernel source 2.6.32-5-vserver-amd64
kernel destination 2.6.22.18-vs2.2.0.6

Hardware source: Quad-Core AMD Opteron(tm) Processor 2378
hardware destination: Intel(R) Pentium(R) 4 CPU 3.20GHz

df -hi reports plenty of free inodes. Source uses from 1-2%, destination 1%. There is also several GB left.

Setup: Source is the new server, destination was the old server which is now redelegated to hotspare (soon to be offsite) to avoid having to restore from backup if something bad happens.

Will try upgrading destination to squeeze. Upgrading did not work, same result.

rsync command running as root rsync -a -v -P --stats --numeric-ids --exclude=/dev --exclude=/proc --exclude=/sys root@<servername>:/* .
Comment 8 Jon Bendtsen 2011-02-18 07:46:34 UTC
Some old unchanged files are transfered again, the sysadm/Guide ...
files and directories are not transfered?

data/cvs/sysadm/Guide to Partner access a.doc,v
     1149519 100%   88.75MB/s    0:00:00 (xfer#4046, to-check=1009/316808)
data/cvs/sysadm/Active-Directory-sync/
rsync: recv_generator: mkdir "/mnt/data/cvs/sysadm/Active-Directory-sync" failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
data/cvs/sysadm/cvsmail/
rsync: recv_generator: mkdir "/mnt/data/cvs/sysadm/cvsmail" failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
rsync: mkstemp "/mnt/data/cvs/sysadm/.Guide to Partner access a.doc,v.UpUy7A" failed: No space left on device (28)
Comment 9 Jon Bendtsen 2011-02-18 08:14:29 UTC
Running xfs_info on my destination filesystem gives me:

meta-data=/dev/mapper/VG-<name>   isize=256    agcount=5, agsize=91750400 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=419430400, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0
Comment 10 Jon Bendtsen 2011-02-21 04:54:20 UTC
I reformated my LV and transfered the files again over the weekend. It works fine. Even when I rerun rsync monday morning. It also works after extending the XFS filesystem. The difference could be that this time the filesystem did not run completely out of space during the rsync. It does not matter if I push or pull using rsync.