Bug 10159 - Windows 8 and Windows 2012 Backup fails against Samba 4.0
Windows 8 and Windows 2012 Backup fails against Samba 4.0
Status: NEW
Product: Samba 4.0
Classification: Unclassified
Component: File services
4.0.9
All All
: P5 normal
: ---
Assigned To: Samba QA Contact
Samba QA Contact
:
: 10055 10296 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-19 22:59 UTC by Andrew Bartlett
Modified: 2016-04-28 16:10 UTC (History)
15 users (show)

See Also:


Attachments
Wireshark capture, max protocol = SMB2_10 (101.40 KB, application/x-xz)
2013-09-19 23:09 UTC, Justin Maggard
no flags Details
Wireshark capture, max protocol = SMB2_02 (107.75 KB, application/x-xz)
2013-09-19 23:10 UTC, Justin Maggard
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Bartlett 2013-09-19 22:59:56 UTC
The tool is 'windows server backup' on windows 2012

Setting 'max protocol = smb2_02' works, but 'max protocol = smb2_10' (and newer) fails. 

We noticed that when failing a ioctl with smb2 we returned NT_STATUS_INVAID_DEVICE_REQUEST, not NT_STATUS_NOT_SUPPORTED per 
MS-SMB2 3.3.5.15.9 Handling a Resiliency Request, however changing this in smb_ioctl.c didn't help.
Comment 1 Justin Maggard 2013-09-19 23:09:47 UTC
Created attachment 9232 [details]
Wireshark capture, max protocol = SMB2_10
Comment 2 Justin Maggard 2013-09-19 23:10:15 UTC
Created attachment 9233 [details]
Wireshark capture, max protocol = SMB2_02
Comment 3 Justin Maggard 2013-09-19 23:12:44 UTC
This was the patch we tried that didn't help.

--- samba4-4.0.9.orig/source3/smbd/smb2_ioctl.c
+++ samba4-4.0.9/source3/smbd/smb2_ioctl.c
@@ -604,8 +604,6 @@ static struct tevent_req *smbd_smb2_ioct
 		if (NT_STATUS_EQUAL(status, NT_STATUS_NOT_SUPPORTED)) {
 			if (IS_IPC(smbreq->conn)) {
 				status = NT_STATUS_FS_DRIVER_REQUIRED;
-			} else {
-				status = NT_STATUS_INVALID_DEVICE_REQUEST;
 			}
 		}
Comment 4 Andrew Bartlett 2013-09-23 22:34:14 UTC
The 'failing' frames seem to be 1689 and 1690 in the SMB2_10 trace
Comment 5 Jeremy Allison 2013-09-26 16:41:40 UTC
Yeah - problem is we weren't planning to do resilient file handles (they have to be requested by applications, plus they're a royal pain :-).

I bet MS has never tested their code against a server that reports SMB 2.10 but doesn't support resilient handles.

I'll look into the protocol specs for more details. I'm in Europe at the moment - back on 30th Sept. Maybe I can come round to debug ?

Jeremy.
Comment 6 Justin Maggard 2013-09-26 16:46:04 UTC
Sure, that works for me.  I'll be around all week.
Comment 7 Andrew Bartlett 2013-09-26 16:46:21 UTC
(In reply to comment #5)
> Yeah - problem is we weren't planning to do resilient file handles (they have
> to be requested by applications, plus they're a royal pain :-).
> 
> I bet MS has never tested their code against a server that reports SMB 2.10 but
> doesn't support resilient handles.
> 
> I'll look into the protocol specs for more details. I'm in Europe at the moment
> - back on 30th Sept. Maybe I can come round to debug ?
> 
> Jeremy.

That is exactly the situation.  I've raised it in person and I'm working on this with Microsoft.  I'm going to get their official position before I leave the event this week. 

We are also going to double-check the ioctl reponse compared to the spec.
Comment 8 Jeremy Allison 2013-09-27 09:11:44 UTC
Thanks Andrew. If you can find out from MS what we should do in this situation (or if there's anything we can do except drop the protocol level) that would be really helpful.

Jeremy.
Comment 9 Andrew Bartlett 2013-09-27 16:55:47 UTC
(In reply to comment #8)
> Thanks Andrew. If you can find out from MS what we should do in this situation
> (or if there's anything we can do except drop the protocol level) that would be
> really helpful.
> 
> Jeremy.

Yep, that's exactly what we are doing, but seems unlikely at this point, beyond implementing resilient file handles.  I'll get an agreed statement explaining things in detail today.

The Microsoft case number is 113092710818901.  If this impacts on their customers and our users with a Premier support arrangement with Microsoft, then this number should be quoted to avoid duplicate investigations. 

For the curious, the very confused threads in the Microsoft forums are:
http://answers.microsoft.com/en-us/windows/forum/windows_8-performance/error-in-backup-0x807800c5/28234fd0-6116-4ba2-afef-cbab00e6a6a9

At the bottom of this page you can see a reasonable summary of what the view is from our users:
http://answers.microsoft.com/en-us/windows/forum/performance/backup-error-0x807800c5/95959b0d-8f2b-4bcb-adb7-ecac48b6c239
Comment 10 Andrew Bartlett 2013-09-27 19:45:41 UTC
So, this is the situation, as we understand it, on the Windows side of this:

HyperV has been possible (but unsafe) on SMB1 and early SMB2 versions, but became supportable with SMB3.  To make it supportable, resilient handles became a requirement.  

The unfortunate thing in this case is to make this supportable, that is to ensure that a failure (for some reason) to get resilient handle was not silently ignored (posing data corruption risks) all SMB3 servers were assumed (and asserted) to support resilient handles. 

This ensures that known configurations - such as Windows / Windows, which would always negotiate SMB3, would either work safely, or fail hard.  That is, the HyperV code actually checks for the SMB2_02, as we suspected.  (That is, versions higher than are asserted to have resilient handles)

Why this matters is because Windows Backup uses a code path in HyperV to handle the full system backup.  

Microsoft is investigating how to proceed with restoring the previous behaviour of Windows Backup. 

As perceived from the Backup user perspective however, this went from 'worked with a SMB2 NAS' in Windows 7, to 'fails with an SMB2 NAS' with Windows 8, because nothing in the error messages makes this clear.
Comment 11 Jeremy Allison 2013-09-28 09:37:34 UTC
Hmmm. What would happen if we just lied and said we did resilient handles (with a horrid "fake resilient handles" parameter) ? I doubt the backup software actually *needs* this feature (I wonder why it is requesting it ?).

Jeremy.
Comment 12 Andrew Bartlett 2013-09-28 15:00:09 UTC
I did consider such an approach. 

The reason I would be oppose it is that once it enters the Samba meme, folks will start setting it, because they saw it in someone's posted smb.conf that they copied.  Then NAS vendors who get complaints about backup start setting it, not just for customers specifically during support incidents (just as they set 'max protocol' to solve this for now), but generally.  Then someone runs HyperV against such a NAS, an admin trips over a network cable, and the world goes to mush...
Comment 13 Andrew Bartlett 2013-09-28 15:07:21 UTC
I should also note that my discussions with Microsoft folks did not indicate that Backup (as currently constructed) had any method to indicate that it did not need this feature.  The suggestion at this point is that the shared code path with HyperV unilaterally requires resiliency, and that this is intentional (to make HyperV stable and supportable over SMB3).

Microsoft is continuing to investigate their and our options here.
Comment 14 Andrew Bartlett 2013-11-15 19:46:22 UTC
Bug #10055 shows an interesting but different facet to this problem.  Apparently this also impacts simply mounting an disk image (what on linux would be a loop device mount) over CIFS.

Also, Bug #9938 shows the intended behaviour of HyperV, that is requiring this ioctl for SMB2.10 and above.
Comment 15 Jason 2013-11-28 05:51:40 UTC
*** Bug 10296 has been marked as a duplicate of this bug. ***
Comment 16 Andrew Bartlett 2014-02-12 21:19:25 UTC
*** Bug 10055 has been marked as a duplicate of this bug. ***
Comment 17 ATechGuy 2014-03-19 12:56:20 UTC
(In reply to comment #14)
> Bug #10055 shows an interesting but different facet to this problem. 
> Apparently this also impacts simply mounting an disk image (what on linux would
> be a loop device mount) over CIFS.
> 
> Also, Bug #9938 shows the intended behaviour of HyperV, that is requiring this
> ioctl for SMB2.10 and above.


These two bugs (10159, 9938) are the same, but manifest themselves using different application on the Windows side. (backup, Hyper-v/Mount) They both have to do with windows expecting MORE from a CIFS server when it reports itself being 3.0 (perhaps 2.10+). The bottom line is that if Samba wants to report itself as being 3.0 compatible, it must FULLY comply with the expectations of a 3.0 server and not offer 90% of the package.

Whether or not Windows actually USES the extra 10% functionality is not the issue here. And "faking it" while clever, is clearly not the desired path.

IMHO, a new bug number, collecting all the MS apps that do different things when connected to a SMB 2.10+/3.0 server should be listed, so we don't keep creating multiple bug reports, and in this bug it needs to be noted what deficiencies are in Samba 4 when representing an SMB level greater than 2_02.
Comment 18 Andrew Bartlett 2014-03-19 22:22:15 UTC
(In reply to comment #17)
> (In reply to comment #14)
> > Bug #10055 shows an interesting but different facet to this problem. 
> > Apparently this also impacts simply mounting an disk image (what on linux would
> > be a loop device mount) over CIFS.
> > 
> > Also, Bug #9938 shows the intended behaviour of HyperV, that is requiring this
> > ioctl for SMB2.10 and above.
> 
> 
> These two bugs (10159, 9938) are the same, but manifest themselves using
> different application on the Windows side. (backup, Hyper-v/Mount) They both
> have to do with windows expecting MORE from a CIFS server when it reports
> itself being 3.0 (perhaps 2.10+). 

The problem is a little more subtle than this.  

Even when KB 2920193 from Microsoft is applied, Windows 8 backup still fails, so we need this bug open.  Frustratingly, the remaining issue is different to the ISO mount issue!

That is why bug #10055 is closed, because there is a workaround, but this is not, because the backup issue isn't fixed.

Bug #9938 is different again, we are told that this won't be fixed on the Microsoft side (it is deliberate), and we will close that when we have the FSCTL_LMR_REQUEST_RESILIENCY feature. 

I hope this clarifies things.

Andrew Bartlett
Comment 19 Andrew Bartlett 2014-05-02 04:17:10 UTC
Windows 8.1 patch 1 should have a fix for this (relaxing the need for this feature for backup and .iso mounting).  It would be helpful if this was verified.
Comment 20 guitman423 2014-09-26 15:11:18 UTC
The Windows 8.1/2012 patch (KB2919355) does not fix the issue with Hyper V. I still receive: "Remote SMB share does not support resiliency." when moving from local storage to a non-Microsoft SMB 3 share.