When running Windows backup against a samba share, the operation fails right after creation of the volume snapshot with message "Incorrect function"

This happens ONLY against Samba 4.3.x series and ONLY on Windows 8.1, Windows server 2012R2, Windows 10, and Windows Server 2016 Technical Preview.

To reproduce, set up a stand-alone Samba 4.3.x server according to https://wiki.samba.org/index.php/Standalone_server and attempt to create a system image of a Windows machine on a share of that server.

This bug is different from https://bugzilla.samba.org/show_bug.cgi?id=10159. In the other bug, the root cause was Samba's lack of support for resilient file handles, and is fixed or has fixes on the Windows side (a hotfixed Windows 8, an up-to-date Win8.1, and Win10 can backup without resiliency support). This bug occurs when the SMB dialect is 3.0.2 or 3.1.1 and the client machine opens the .vhdx file with SVHDX_OPEN_DEVICE_CONTEXT or SVHDX_OPEN_DEVICE_CONTEXT_V2 create context.

If the client is an older version of Windows, the dialect is older - bug doesn't happen.

If samba is older, it ignores the SVHDX_OPEN_DEVICE_CONTEXT which turns out to be the correct thing to do.

A workaround is to set "server max protocol" to SMB3_00 or lower.

An excerpt of a backup command line session on a Win10 client:

C:\WINDOWS\system32>wbadmin start backup -allCritical -backupTarget:\\192.168.40.251\ush -user:localu1 -password:password00
wbadmin 1.0 - Backup command-line tool
(C) Copyright 2013 Microsoft Corporation. All rights reserved.


Note: The backed up data cannot be securely protected at this destination.
Backups stored on a remote shared folder might be accessible by other
people on the network. You should only save your backups to a location
where you trust the other users who have access to the location or on a
network that has additional security precautions in place.

Retrieving volume information...
This will back up System Reserved (500.00 MB),(C:) to \\192.168.40.251\ush.
Do you want to start the backup operation?
[Y] Yes [N] No y

The backup operation to \\192.168.40.251\ush is starting.
Creating a shadow copy of the volumes specified for backup...
Creating a shadow copy of the volumes specified for backup...
Creating a shadow copy of the volumes specified for backup...
Creating a shadow copy of the volumes specified for backup...
The backup of volume System Reserved (500.00 MB) could not be completed. Error: Incorrect function.
Summary of the backup operation:
------------------

The backup operation stopped before completing.
The backup operation stopped before completing.
Detailed error: Incorrect function.
Log of files successfully backed up:
C:\WINDOWS\Logs\WindowsBackup\Backup-23-02-2016_18-30-14.log

Log of files for which backup failed:
C:\WINDOWS\Logs\WindowsBackup\Backup_Error-23-02-2016_18-30-14.log

There was a failure in preparing the backup image of one of the volumes in the backup set.
Incorrect function.

C:\WINDOWS\system32>type c:\Windows\Logs\WindowsBackup\Backup_Error-23-02-2016_14-31-49.log
Backup of volume \\?\Volume{1948156d-0000-0000-0000-100000000000}\ has failed. Incorrect function.