Bug 12033 - smbd should support copy_file_range() for FSCTL_SRV_COPYCHUNK
Summary: smbd should support copy_file_range() for FSCTL_SRV_COPYCHUNK
Status: ASSIGNED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: File services (show other bugs)
Version: unspecified
Hardware: All All
: P5 enhancement (vote)
Target Milestone: ---
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL:
Keywords:
: 12785 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-07-25 09:29 UTC by Björn Jacke
Modified: 2021-07-19 17:39 UTC (History)
6 users (show)

See Also:


Attachments
patch for master (5.61 KB, patch)
2017-02-09 08:13 UTC, Björn Jacke
no flags Details
waf check only part (1.13 KB, patch)
2020-01-14 09:46 UTC, Björn Jacke
no flags Details
Patch for 4.14 cherry-picked from master (19.17 KB, patch)
2021-07-01 15:34 UTC, Ralph Böhme
jra: review-
Details
Patch for 4.13 backported from master (19.19 KB, patch)
2021-07-01 15:35 UTC, Ralph Böhme
jra: review+
Details
Patch for 4.13 backported from master (19.19 KB, patch)
2021-07-01 15:35 UTC, Ralph Böhme
no flags Details
Patch for 4.14 cherry-picked from master (20.50 KB, patch)
2021-07-02 03:15 UTC, Ralph Böhme
jra: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Björn Jacke 2016-07-25 09:29:02 UTC
copy_file_range is supported since kernel 4.5. We should check if this call is supported and use it for server side copychung operation.

Interesting also for Kernel 4.7, where the NFS client code also supports that call and can speed up Samba servers with NFS file stores.

Side note: if copy_file_range support is found, btrfs should not override the copy_chunk functions but use the generic code.
Comment 1 Björn Jacke 2016-09-07 09:35:54 UTC
with linux kernel 4.8 even xfs will profit immensely from the use of copy_file_range with the introduction of reflink copies.
Comment 2 Björn Jacke 2017-02-09 08:13:29 UTC
Created attachment 12912 [details]
patch for master
Comment 3 Ralph Böhme 2017-06-02 14:17:48 UTC
As just discussed on the phone:

* in vfs_default.c:vfswrap_copy_chunk_send() we should use copy_file_range() if flags does not contain VFS_COPY_CHUNK_FL_MUST_CLONE

* if flags contains VFS_COPY_CHUNK_FL_MUST_CLONE we must use the IOC_CLONE_RANGE ioctl()

* we then can remove the copy_chunk handler from vfs_btrfs
Comment 4 Björn Jacke 2020-01-14 09:46:53 UTC
Created attachment 15729 [details]
waf check only part
Comment 5 Björn Jacke 2020-01-14 09:47:20 UTC
Ralph: some weeks after my patch you made all that copy chunk async, which requires this patch to be reworked for this. Can you have a look at his, I actually don't completely get throug the new code to adopt the patch myself.

I also think that for local copies the copy chunk operation should be not split into 16MiB small pieces if this is possible because even a local copy of *any* file size is an atomic operation with most filesystems these days with copy_file_range. Maybe COPYCHUNK_MAX_TOTAL_LEN can be at least multiplied by 10 or so to make copies of very large files not too fragmented (and with unneeded overhead).
Comment 6 Ralph Böhme 2021-06-14 15:02:59 UTC
*** Bug 12785 has been marked as a duplicate of this bug. ***
Comment 7 David Disseldorp 2021-06-15 08:18:17 UTC
(In reply to Björn Jacke from comment #5)
...
> I also think that for local copies the copy chunk operation should be not
> split into 16MiB small pieces if this is possible because even a local copy
> of *any* file size is an atomic operation with most filesystems these days
> with copy_file_range.

Chunking large requests makes sense to me. copy_file_range() is not atomic. It *may* result in a reflink / clone on XFS and Btrfs, may be handled as splice / sendfile IO or as an offloaded copy for network filesystems.
Comment 8 Ralph Böhme 2021-06-25 14:31:09 UTC
(In reply to David Disseldorp from comment #7)
David, I have a WIP patchset that adds copy_file_range() support here: <https://gitlab.com/samba-team/samba/-/merge_requests/2044>.

The final patch removes BTRFS_IOC_CLONE_RANGE support from vfs_btrfs and I wanted to check back with you if you think we're ready for that jump?

In theory an older kernel could lack both the copy_file_range() function and the raw syscall, BUT could have btrfs version that supports BTRFS_IOC_CLONE_RANGE. I don't know how common such a combination can be.

Thoughts?
Comment 9 Samba QA Contact 2021-06-30 17:41:03 UTC
This bug was referenced in samba master:

4dcc04228df233be15efe9c31bcbaba822b140c4
e2d524d4baee193b0d03df3e7edda48b3cb4bddf
4f1a02909b8694dcc30fd5c7c6772fcfa1092ed9
e72be5213335ab1ea0f9f396ab071669231c151b
accaa2f1f67a7f064a4ce03a120d7b2f8e847ccf
Comment 10 Ralph Böhme 2021-07-01 15:34:55 UTC
Created attachment 16669 [details]
Patch for 4.14 cherry-picked from master
Comment 11 Ralph Böhme 2021-07-01 15:35:15 UTC
Created attachment 16670 [details]
Patch for 4.13 backported from master
Comment 12 Ralph Böhme 2021-07-01 15:35:16 UTC
Created attachment 16671 [details]
Patch for 4.13 backported from master
Comment 13 Jeremy Allison 2021-07-01 16:46:01 UTC
Comment on attachment 16669 [details]
Patch for 4.14 cherry-picked from master

These need the follow-up patch changing pathref -> io. Sorry for missing that in the original review.
Comment 14 Jeremy Allison 2021-07-01 16:55:29 UTC
Comment on attachment 16670 [details]
Patch for 4.13 backported from master

LGTM. No pathref functions here :-).
Comment 15 Ralph Böhme 2021-07-01 17:02:10 UTC
(In reply to Jeremy Allison from comment #13)
Ah, drat, that is *exactly* what I wanted to do, but I got confused...
Comment 16 Samba QA Contact 2021-07-01 17:46:05 UTC
This bug was referenced in samba master:

0e3ddc27ed6d603a21cb2b187f3295506d560604
Comment 17 Ralph Böhme 2021-07-02 03:15:05 UTC
Created attachment 16672 [details]
Patch for 4.14 cherry-picked from master

This one includes the additional fix. Sorry for missing this in the initial backport!
Comment 18 Jeremy Allison 2021-07-02 17:07:23 UTC
Comment on attachment 16672 [details]
Patch for 4.14 cherry-picked from master

LGTM. Thanks.
Comment 19 Jeremy Allison 2021-07-02 17:07:31 UTC
Comment on attachment 16672 [details]
Patch for 4.14 cherry-picked from master

LGTM. Thanks.
Comment 20 Jeremy Allison 2021-07-02 17:08:08 UTC
Re-assigning to Karolin for inclusion in 4.14.next.
Comment 21 Karolin Seeger 2021-07-19 06:33:23 UTC
(In reply to Jeremy Allison from comment #20)
Pushed to autobuild-v4-14-test.
Just to make sure: This one should not go into 4.13?
Comment 22 Jeremy Allison 2021-07-19 17:39:09 UTC
That's correct, 4.13.x doesn't have fsp_get_io_fd().