Bug 10170 - rsync should support reflink similar to cp --reflink
rsync should support reflink similar to cp --reflink
Status: NEW
Product: rsync
Classification: Unclassified
Component: core
All All
: P5 enhancement
: ---
Assigned To: Wayne Davison
Rsync QA Contact
: 9041 (view as bug list)
Depends on:
  Show dependency treegraph
Reported: 2013-09-27 00:25 UTC by samba
Modified: 2015-10-02 13:52 UTC (History)
6 users (show)

See Also:

prototype patch for --reflink support (17.67 KB, patch)
2015-01-06 20:32 UTC, David Taylor
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description samba 2013-09-27 00:25:30 UTC
new filesystems support the reflink system call...
the standard GNU cp utility now supports this too.
this functionality is invaluable for future rsync.
@BasketCase recommended I write this as a feature.

Here is some background reference:

Thanks in advance, happy to answer any questions I can!

Comment 1 Kevin Korb 2013-09-27 00:31:00 UTC
*** Bug 9041 has been marked as a duplicate of this bug. ***
Comment 2 David Taylor 2015-01-06 20:32:31 UTC
Created attachment 10585 [details]
prototype patch for --reflink support

I have written a rough draft of a patch to enable support for (Btrfs) reflinks in rsync.  It is very lightly tested.

Only Btrfs is supported (using BTRFS_IOC_CLONE), and this is intended to provide a better alternative to --inplace for updating a backup which will be snapshotted repeatedly.  It avoids the problems of --inplace with
partial updates and stale hardlinks.

It adds two new options:


When updating an existing file, rather than creating an entirely new temporary file, rsync will create a 'reflink' copy of the original file and update it in place.  When creating a backup copy of a file, rsync will also create a 'reflink' copy rather than rewriting the file.  If a 'reflink' copy cannot be performed it will fall back to the normal, non-reflink behaviour.

--reflink-always behaves like --reflink, but disables the fallback if a reflink fails.
Comment 3 roland 2015-01-06 20:58:30 UTC
fantastic !

did not know about that feature in btrfs, but as i already did inplace backups with rsync on btrfs, i think i will give that a try.

i just wondered about reflink support in zfs and putting a link here for reference: https://github.com/zfsonlinux/zfs/issues/405
Comment 4 James 2015-01-09 03:16:05 UTC
(In reply to David Taylor from comment #2)

Sweet... Glad to hear someone is hacking on this.
Comment 5 kdave 2015-03-27 01:40:43 UTC
(In reply to David Taylor from comment #2)

A few things:

* thanks for working on reflink support in rsync :)

* btrfs is not the only filesystem to support reflink, ocfs2 does as well, so you might make the formulations more generic

* for btrfs, there are 2 types of the cloning ioctl:
 1) that does file-to-file clone (same for ocfs2), IOC_CLONE
 2) range cloning, IOC_CLONE_RANGE

I believe for rsync option 2 should be implemented in the similar way --inplace works.

* the proposed patch does not work in all scenarios:

$ btrfs subvol create subv1
$ btrfs subvol create subv2
$ <create big file subv1/testfile>
$ rsync --reflink subv1/testfile subv2

according to strace, rsync does not call the clone ioctl at all (none of the forked processes)

$ rsync --reflink-always subv1/testfile subv2
    240,475,844 100%   43.54MB/s    0:00:05 (xfr#1, to-chk=0/1)
rsync: open "/subv2/testfile": No such file or directory (2)
rsync: reflink of "/subv2/.testfile.8bxNk0" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.2dev]

and the target file is not created, the same command without --reflink works.

* for speed comparison, I did 'cp --reflink subv1/testfile subv2' and it takes near to no time, compared to 5 seconds for bare copy, I'm expecting that rsync --reflink would take comparable time to cp+reflink