Given enough hard links in a source folder and a small enough destination volume, copying with rsync -- hard-links can fail whereas the Mac OS X ditto program succeeds. Rsync fails by exhausting the maximum number of hard links on the destination. I have made a script that demonstrates this bug. It makes two minimal ram disks, with a volume on each. On the source volume it makes a maximal set of hard linked files that can be copied correctly to the other volume using ditto but not using rsync.
Created attachment 5956 [details] test script that demonstrates the bug
test script output looks like this rsync: mkstemp "/Volumes/Copy/links/03/29/.83b.QlmOQ8" failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.84b.KFW6da" failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.85b.vB6pC5" failed: No space left on device (28) rsync: link "/Volumes/Copy/links/03/28/36a" => links/03/28/36b failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.86b.jBXuVJ" failed: No space left on device (28) rsync: link "/Volumes/Copy/links/03/28/37a" => links/03/28/37b failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.87b.C3svVR" failed: No space left on device (28) rsync: link "/Volumes/Copy/links/03/28/38a" => links/03/28/38b failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.88b.RcRxZH" failed: No space left on device (28) rsync: link "/Volumes/Copy/links/03/28/39a" => links/03/28/39b failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.89b.mFP87z" failed: No space left on device (28) rsync: link "/Volumes/Copy/links/03/28/40a" => links/03/28/40b failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.90b.GQbvqB" failed: No space left on device (28) rsync: link "/Volumes/Copy/links/03/28/41a" => links/03/28/41b failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.91b.L5bJ8n" failed: No space left on device (28) rsync: link "/Volumes/Copy/links/03/28/42a" => links/03/28/42b failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.92b.PXp0X0" failed: No space left on device (28) rsync: link "/Volumes/Copy/links/03/28/43a" => links/03/28/43b failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.93b.vpOJOO" failed: No space left on device (28) rsync: link "/Volumes/Copy/links/03/28/44a" => links/03/28/44b failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.94b.8Y8sFs" failed: No space left on device (28) rsync: link "/Volumes/Copy/links/03/28/45a" => links/03/28/45b failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.95b.41bN0l" failed: No space left on device (28) rsync: mkstemp "/Volumes/Copy/links/03/29/.96b.rMQnZB" failed: No space left on device (28) ...
Is this really an rsync bug? I experienced the same thing on Linux with ext3 with our backup system, which is heavily dependent upon --link-dest. I've tested switching to xfs, and this solves the problem because the real issue isn't rsync but instead the underlying file system. I also confirmed that cp -Rl had the same problem and solution. http://www.mail-archive.com/rsync@lists.samba.org/msg25950.html I did wonder whether rsync should, when faced with this error, fall back on creating a copy. But should rsync include behavior that exists only to work around a file system limit? Perhaps only as a command line option (ie. definitely not the default behavior)? I'm also not sure that it should always do this. It's not so bad to do so for --link-dest, which is really just an optimization. But if rsync is to be making a reliable copy of the source at the destination, this should include mimicry of any links. In that case, falling back to copy would be a bad idea, I believe.
On Mac OS X, the ditto program works where rsync fails, so it seems this is a problem with rsync (and now we know it is also a problem with "cp -Rl"). Wayne, perhaps you could ask a friend at Apple how ditto does the equivalent of --hard-links.
Why is this not a blocker? Why is it still marked New?
(In reply to comment #5) > Why is this not a blocker? It does not "block development and/or testing work" (https://bugzilla.samba.org/page.cgi?id=fields.html#bug_severity). An example of a blocker would be if rsync were unable to add directories to the file list at all; then very little meaningful testing could be done. > Why is it still marked New? No one has started to work on it.
I experience the same problem on Linux rsync'ing over the network. Both hosts uses XFS filesystem(s), both uses Volumegroups that has been lvextended and then xfs_growfs. Source uses several smaller LV in different VG (some SSD, some HDD), all XFS and mounted in various locations. Destination uses 1 big LV in 1 VG. rsync source: tried both debian lenny and squeeze packages, 3.0.3-2 and 3.0.7-2. rsync destination debian lenny package, 3.0.3-2 Kernel source 2.6.32-5-vserver-amd64 kernel destination 2.6.22.18-vs2.2.0.6 Hardware source: Quad-Core AMD Opteron(tm) Processor 2378 hardware destination: Intel(R) Pentium(R) 4 CPU 3.20GHz df -hi reports plenty of free inodes. Source uses from 1-2%, destination 1%. There is also several GB left. Setup: Source is the new server, destination was the old server which is now redelegated to hotspare (soon to be offsite) to avoid having to restore from backup if something bad happens. Will try upgrading destination to squeeze. Upgrading did not work, same result. rsync command running as root rsync -a -v -P --stats --numeric-ids --exclude=/dev --exclude=/proc --exclude=/sys root@<servername>:/* .
Some old unchanged files are transfered again, the sysadm/Guide ... files and directories are not transfered? data/cvs/sysadm/Guide to Partner access a.doc,v 1149519 100% 88.75MB/s 0:00:00 (xfer#4046, to-check=1009/316808) data/cvs/sysadm/Active-Directory-sync/ rsync: recv_generator: mkdir "/mnt/data/cvs/sysadm/Active-Directory-sync" failed: No space left on device (28) *** Skipping any contents from this failed directory *** data/cvs/sysadm/cvsmail/ rsync: recv_generator: mkdir "/mnt/data/cvs/sysadm/cvsmail" failed: No space left on device (28) *** Skipping any contents from this failed directory *** rsync: mkstemp "/mnt/data/cvs/sysadm/.Guide to Partner access a.doc,v.UpUy7A" failed: No space left on device (28)
Running xfs_info on my destination filesystem gives me: meta-data=/dev/mapper/VG-<name> isize=256 agcount=5, agsize=91750400 blks = sectsz=512 attr=2 data = bsize=4096 blocks=419430400, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=32768, version=2 = sectsz=512 sunit=0 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0
I reformated my LV and transfered the files again over the weekend. It works fine. Even when I rerun rsync monday morning. It also works after extending the XFS filesystem. The difference could be that this time the filesystem did not run completely out of space during the rsync. It does not matter if I push or pull using rsync.