Bug 15050 - smbclient no longer can connect to Azure
Summary: smbclient no longer can connect to Azure
Status: ASSIGNED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: libsmbclient (show other bugs)
Version: unspecified
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Samba QA Contact
QA Contact: Samba QA Contact
URL: https://gitlab.com/samba-team/samba/-...
Keywords:
Depends on:
Blocks:
 
Reported: 2022-04-21 14:33 UTC by falk0069
Modified: 2022-05-02 20:14 UTC (History)
1 user (show)

See Also:


Attachments
wireshark smbclient trace failing between ubuntu and Azure (1.19 KB, image/CAP)
2022-04-21 14:33 UTC, falk0069
no flags Details
working mount.cifs comparison (11.77 KB, application/octet-stream)
2022-04-21 14:45 UTC, falk0069
no flags Details
smbclient run output after some code updates (24.14 KB, text/plain)
2022-04-27 15:32 UTC, falk0069
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description falk0069 2022-04-21 14:33:55 UTC
Created attachment 17273 [details]
wireshark smbclient trace failing between ubuntu and Azure

Starting Early April 2022, the smbclient no longer can connect to Azure and reports this error:
protocol negotiation failed: NT_STATUS_INVALID_NETWORK_RESPONSE

The problem is reproduced on all these versions:
Version 4.5.5
Version 4.10.16
Version 4.13.17-Ubuntu
Version 4.17.0pre1-DEVELOPERBUILD

Tracing the code, I find the problem at this point:
./libcli/smb/smbXcli_base.c
static void smbXcli_negprot_smb2_done(struct tevent_req *subreq)
{
...
        conn->smb2.server.capabilities  = IVAL(body, 24);
        conn->smb2.server.max_trans_size= IVAL(body, 28);
        conn->smb2.server.max_read_size = IVAL(body, 32);
        conn->smb2.server.max_write_size= IVAL(body, 36);
        conn->smb2.server.system_time   = BVAL(body, 40);
        conn->smb2.server.start_time    = BVAL(body, 48);

        security_offset = SVAL(body, 56);
        security_length = SVAL(body, 58);

        if (security_offset != SMB2_HDR_BODY + iov[1].iov_len) {
                tevent_req_nterror(req, NT_STATUS_INVALID_NETWORK_RESPONSE);
                return;
        }

The problem is the security_offset and security_length are zero.

It would appear Azure is not send the Security Blob in the Negotiate Protocol Response.  Here is a snippet from WireShark (attached):

        Security mode: 0x03, Signing enabled, Signing required
        Dialect: SMB 3.1.1 (0x0311)
        NegotiateContextCount: 2
        Server Guid: 294a5004-d627-4bc0-9342-726e72993070
        Capabilities: 0x00000017, DFS, LEASING, LARGE MTU, PERSISTENT HANDLES
        Max Transaction Size: 1048576
        Max Read Size: 1048576
        Max Write Size: 1048576
        Current Time: Apr 20, 2022 15:39:48.378980600 Central Daylight Time
        Boot Time: No time specified (0)
        Blob Offset: 0x00000000
        Blob Length: 0
        Security Blob: <MISSING>: NO DATA
        NegotiateContextOffset: 0x00000080
 

It is also worth mentioning that mount.cifs is capable to mount the Azure share.  It seems to start out with a 'Session Setup Request' message first instead of 'Negotiate Protocol Request'.  Not sure if that makes a difference or not.  I can share a trace of that as well if useful.
Comment 1 falk0069 2022-04-21 14:45:35 UTC
Created attachment 17274 [details]
working mount.cifs comparison
Comment 2 falk0069 2022-04-21 14:48:29 UTC
I attached a working mount.cifs trace as well.  The one I just generated from my Ubuntu test bed actually does start out with a Negotiate Protocol Request, followed then by the Session Setup Request.  In the attachment we see the Security Blob <MISSING>, but it doesn't seem to cause a problem for mount.cifs.
Comment 3 Jeremy Allison 2022-04-26 00:29:29 UTC
Azure requires encryption. Are you specifying SMB3.1.1 encryption on your command line ?
Comment 4 Stefan Metzmacher 2022-04-26 09:10:57 UTC
The patch from https://gitlab.com/samba-team/samba/-/merge_requests/2497
should fix the problem.
Comment 5 falk0069 2022-04-26 14:25:12 UTC
This is a sample of the command I use:
smbclient //stgsmbtesting.file.core.windows.net/test -A ~/auth.file -m SMB3 -d 10

auth.file:
username = stgsmbtesting
password = jZYb5kloer<truncated>
domain =

I've also tried other versions (e.g. SMB3_11)


I tried the patch but I'm now breaking further down.  I'll play around a bit, but here is a snippet from my gdb session where it is about to die on line 5158:


Breakpoint 3, smbXcli_negprot_smb2_done (subreq=0x5555555c7fd0) at ../../libcli/smb/smbXcli_base.c:5158
5158                    tevent_req_nterror(req, NT_STATUS_INVALID_NETWORK_RESPONSE);
(gdb) l
5153                    return;
5154            }
5155
5156            ctx_ofs = negotiate_context_offset - security_offset;
5157            if (ctx_ofs > iov[2].iov_len) {
5158                    tevent_req_nterror(req, NT_STATUS_INVALID_NETWORK_RESPONSE);
5159                    return;
5160            }
5161            avail = iov[2].iov_len - security_length;
5162            needed = iov[2].iov_len - ctx_ofs;
(gdb) p negotiate_context_offset
$1 = 128
(gdb) p security_offset
$2 = 0
(gdb) p iov[2].iov_len
$3 = 60
(gdb)
Comment 6 Stefan Metzmacher 2022-04-26 16:30:31 UTC
(In reply to Stefan Metzmacher from comment #4)

Ah, ok! Thanks for testing.

I'll create a new patch tomorrow...
Comment 7 falk0069 2022-04-27 15:32:46 UTC
Created attachment 17277 [details]
smbclient run output after some code updates
Comment 8 falk0069 2022-04-27 15:39:44 UTC
(In reply to Stefan Metzmacher from comment #6)
I tried commenting out some more code in the smbXcli_negprot_smb2_done() function.  Basically the section looking at avail vs. needed and all the places using preauth.  Bear in mind I really don't understand what is actually required.  

In any case, I did get further but it still ultimately failed ending in this:

gensec_update_send: spnego[0x5591f2b3bff0]: subreq: 0x5591f2b5c0a0
gensec_update_done: spnego[0x5591f2b3bff0]: NT_STATUS_OK tevent_req[0x5591f2b5c0a0/../../auth/gensec/spnego.c:1631]: state[2] error[0 (0x0)]  state[struct gensec_spnego_update_state (0x5591f2b5c260)] timer[(nil)] finish[../../auth/gensec/spnego.c:2116]
 session setup ok
signed SMB2 message (sign_algo_id=1)
tree connect failed: NT_STATUS_CONNECTION_DISCONNECTED

Attached is the full output.  Note, I have a few extra "DEBUG_PRINTF" lines that I added.

I hope it makes sense to you and have some thoughts on how to make it work with Azure again.

Thanks
Comment 9 Stefan Metzmacher 2022-04-28 15:42:24 UTC
(In reply to falk0069 from comment #8)

Can you try the latest patches I posted to
https://gitlab.com/samba-team/samba/-/merge_requests/2497 ?
Comment 10 Jason 2022-04-28 18:09:02 UTC
The patch works when connecting to Azure file share servers that were returning the 0 length security_offset, and against the file share servers which were still returning a populated security_offset value.
Comment 11 Jason 2022-04-28 18:11:18 UTC
(In reply to Jason from comment #10)
Note that the patches were applied successfully to the samba-4.10.16-18 version which is included in CentOS 7.9.
Comment 12 falk0069 2022-04-28 21:01:57 UTC
(In reply to Stefan Metzmacher from comment #9)

BOOM BABY!

$ ./samba/samba/bin/smbclient //stgsmbtesting.file.core.windows.net/test -A ~/auth.file -m SMB3
Try "help" to get a list of possible commands.
smb: \> ls
  .                                   D        0  Wed Apr 20 21:00:02 2022
  ..                                  D        0  Wed Apr 20 21:00:02 2022
  smb.cap                             A     1838  Wed Apr 20 20:18:06 2022

                83886080 blocks of size 65536. 83886079 blocks available
smb: \>


It seems to be working like a charm.   

Thanks!
Comment 13 falk0069 2022-05-02 14:08:54 UTC
I know this isn't the right place to ask this, but why was the ability to create static binaries removed?  I ended up dropping in your fix to version 4.7 so the '--nonshared-binary=client/smbclient' flag worked when running configure.  If nothing else, just letting others know in case they need to get a quick fix to an isolated system.
Comment 14 Stefan Metzmacher 2022-05-02 14:24:32 UTC
(In reply to falk0069 from comment #13)

In current releases check the following for regressions:
--nonshared-binary=smbtorture,smbd/smbd

Please open a new bug if it doesn't work with client/smbclient
Comment 15 Samba QA Contact 2022-05-02 20:14:03 UTC
This bug was referenced in samba master:

bc22d5ebf928499e8f0b9540721e9a62db029195
8ca99c25bacb6d9b0e6e064b37d6b726d181a487
54c6cf8666b073818301d3a71a37453b44e57b5c