Bug 12380 - vfs_fruit support for Time Machine over SMB
Summary: vfs_fruit support for Time Machine over SMB
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: VFS Modules (show other bugs)
Version: unspecified
Hardware: All All
: P5 enhancement (vote)
Target Milestone: ---
Assignee: Ralph Böhme
QA Contact: Samba QA Contact
Depends on:
Reported: 2016-10-14 02:23 UTC by Kevin
Modified: 2017-10-26 19:53 UTC (History)
10 users (show)

See Also:

Capture from a stock Samba 4.4.6 installation (4.52 MB, application/gzip)
2016-10-14 19:18 UTC, Kevin
no flags Details
Capture from a Samba version running the attached patch for capabilities (15.31 KB, application/gzip)
2016-10-14 19:19 UTC, Kevin
no flags Details
Capture from a working Mac OSX server providing Time capsule services (257.92 KB, application/gzip)
2016-10-14 19:20 UTC, Kevin
no flags Details
WIP: Samba capabilities patch (1.55 KB, patch)
2016-10-14 19:22 UTC, Kevin
no flags Details
Samba 4.5.9+ fullsync advertisement for vfs_fruit.c (1.17 KB, patch)
2017-05-30 14:50 UTC, Chris Roadfeldt
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Kevin 2016-10-14 02:23:38 UTC
Given that Apple has deprecated AFP for Time Machine backups in favor of SMB it would be beneficial to be able to use Samba to provide a network based backup destination.

Apple has published the required capabilities for supporting Time Machine backups over SMB and can be found here: https://developer.apple.com/library/content/releasenotes/NetworkingInternetWeb/Time_Machine_SMB_Spec/

Looking through the code I think this probably fits the best in vfs_fruit.c as that module handles a large amount of other integrations for Mac OSX.

I also have Wireshark captures of a working Time Machine backup to an OSX server and a non-working backup to a Samba server if that would be beneficial.

I started attempting to implement the required features but I would appreciate some help with some of the pieces. 

I was able to change vfs_fruit to advertise the capability correctly but I can not determine if it is possible to flush the data when recieving an SMB2 FLUSH within the VFS layer. Is that possible?

Based on what I can find in the captures I believe that all of the other features are implemented already including SMB2_CREATE_DURABLE_HANDLE_REQUEST_V2 and SMB2_CREATE_REQUEST_LEASE_V2. 

Comment 1 Stefan Metzmacher 2016-10-14 07:53:30 UTC
Please compress the pcap files with gzip --best and attach them to this bug report.

Attaching you work in progress patch would also help.

Comment 2 Ralph Böhme 2016-10-14 08:33:39 UTC
Here's some more stuff that nicely explains what F_FULLSYNC does on OS X:

I remember I simply faked this for Netatalk.
Comment 3 Jeremy Allison 2016-10-14 17:45:41 UTC
Ralph, please ping me about this requirement. I've been talking to Apple directly about this.

Comment 4 Kevin 2016-10-14 19:18:58 UTC
Created attachment 12576 [details]
Capture from a stock Samba 4.4.6 installation
Comment 5 Kevin 2016-10-14 19:19:38 UTC
Created attachment 12577 [details]
Capture from a Samba version running the attached patch for capabilities
Comment 6 Kevin 2016-10-14 19:20:15 UTC
Created attachment 12578 [details]
Capture from a working Mac OSX server providing Time capsule services
Comment 7 Kevin 2016-10-14 19:22:31 UTC
Created attachment 12579 [details]
WIP: Samba capabilities patch

This is a patch to advertise F_FULLSYNC to the client, but it does nothing beyond that at this time.
Comment 8 Kevin 2016-10-16 02:02:44 UTC
I was able to get the patch that I provided to work successfully for Time Machine backups. The reason that it was not working after applying the patch was due to not having durable file handles configured. After adding the following to my smb.conf file I was able to start a Time Machine backup.

- durable handles = yes
- kernel oplocks = no
- kernel share modes = no
- posix locking = no

After the backup I will attempt to restore a file to verify functionality. After reading what Ralph mentioned regarding the F_FULLSYNC fnctrl, my personal opinion is that it is OK to fake it like I did in the patch but I would certainly like some concurrence from others with more knowledge on Samba internals.

Comment 9 Kevin 2016-10-24 00:43:40 UTC
I did some more research on this and I believe that advertising the capability and taking no other action is an acceptable course of action. I went ahead and made it a configurable option, updated the man page, tested the code, and created a Github pull request for this issue. Here is the link to the PR: https://github.com/samba-team/samba/pull/64

Comment 10 Ralph Böhme 2016-11-01 11:18:36 UTC
Thanks for the patches and please keep pushing me to get them integrated. I need to discuss a few technical things with Jeremy related to this before I can start to do a review. So please stay tuned... :)
Comment 11 Kevin 2016-11-03 01:48:56 UTC
Thanks Ralph! I'll ping you in a week through this bug (if you prefer a different interval or communication method just let me know) for a status check/update. In the meantime if you think of anything that can be added or changed just let me know.
Comment 12 Kevin 2016-11-10 23:16:41 UTC
Hi Ralph,
  Have you had a chance to discuss this patch with Jeremy yet?

Comment 13 Nick Tenney 2017-02-01 00:29:19 UTC
User ping on this.  Should I go down the route of applying Kevin's patch for now or await a proper implementation of the feature?  I have been planning to migrate to SMB-based Time Machine backups, but was sorely disappointed to find out about the new fruity extension. :(
Comment 14 Kevin 2017-02-18 23:53:55 UTC
Hi Nick,
  The decision to apply it or not really depends on how much you want to trust your data to it. :) With that being said some changes are being discussed on the mailing list to ensure that backed up data is safe and consistent during events like power offs. I have been running the provided patch for my Mac's on a home server without any issues but it is also not my primary form of backups right now.
Comment 15 Andrew Miskell 2017-03-13 19:36:44 UTC
Another user ping on this as well. Are we any closer to having this functionality put into Samba so it can be used for Time Machine backups?

I have an open enhancement request in the FreeNAS 10 bug tracker to track official support but it's currently pending this enhancement making into into the codebase.
Comment 16 Chris Roadfeldt 2017-05-30 14:50:24 UTC
Created attachment 13242 [details]
Samba 4.5.9+ fullsync advertisement for vfs_fruit.c

Updated patch file for samba 4.5.9+. The fix for bug 12479 introduced changes vfs_fruit.c that broke Kevin's original patch. I've updated the patchfile to apply the fullsync advertisement work around to accomodate.
Comment 17 Kevin 2017-05-30 23:46:44 UTC
Thanks Chris. Just a heads up you can pull an updated patch from https://github.com/samba-team/samba/pull/64.patch which should apply cleanly on master. I keep the pull request updated whenever I notice a breakage or need to make a change. Your patch, while it technically works, looks like it hard codes the option but the updated patch actually makes it a configurable value. That being said, let me know if you notice any breakage with my updated patch.
Comment 18 Chris Roadfeldt 2017-05-31 00:25:52 UTC
Thanks Kevin, I didn't think to look at your PR. I like that approach better than hardcoding it. Will try it out tonight, if there's no reply, then all went well.

For folks who stumble upon this looking for Time Machine, aka TM, support in Samba, I'd go with Kevin's PR patch linked above.
Comment 19 Chris Roadfeldt 2017-05-31 19:22:21 UTC
Having issues with the fedora samba 4.5.10 package with the PR 64 patch. Getting a message from OS X that the samba server doesn't support the required capabilities. Attempted to wireshark the transaction, but not sure what bits I should be looking for. Kevin can you clue me in on the transcation and value I should be looking for? What other info would help?

FYI - I've applied the patch to the rpm build, it applies successfully. I've also configured the TM share with fruit:time machine = yes, removed fruit:fullsync entry. Also removed the duplicate samba config items that fruit:time machine applies as well.
Comment 20 Kevin 2017-05-31 20:52:48 UTC
Chris if you move fruit:time machine to the global configuration section, rather than the per share configuration, does it work then?
Comment 21 Chris Roadfeldt 2017-05-31 21:44:09 UTC
Affirmative, it does work when it's in the global config.

Working config, global config.

fruit:aapl = yes
fruit:time machine = yes

Previous, non working config per share.

[time machine]
fruit:aapl = yes
fruit:time machine = yes
Comment 22 Ralph Böhme 2017-10-26 19:53:41 UTC
This will ship with Samba 4.8.