Bug 10958 - Session expired - reconnect - access denied
Session expired - reconnect - access denied
Status: RESOLVED FIXED
Product: Samba 4.1 and newer
Classification: Unclassified
Component: AD: LDB/DSDB/SAMDB
unspecified
x64 Linux
: P5 critical
: ---
Assigned To: Karolin Seeger
Samba QA Contact
:
Depends on:
Blocks: 10077
  Show dependency treegraph
 
Reported: 2014-11-24 13:12 UTC by Kyriacos Georgiou
Modified: 2014-12-20 15:52 UTC (History)
3 users (show)

See Also:


Attachments
wireshark packet capture (46.98 KB, application/octet-stream)
2014-11-25 15:01 UTC, Kyriacos Georgiou
no flags Details
server signing mandatory (5.91 KB, application/octet-stream)
2014-12-11 10:46 UTC, Kyriacos Georgiou
no flags Details
server signing mandatory and client signing required (2.90 KB, application/octet-stream)
2014-12-11 10:47 UTC, Kyriacos Georgiou
no flags Details
packet capture after reg key set to enforce signing on client (76.21 KB, application/octet-stream)
2014-12-11 11:39 UTC, Kyriacos Georgiou
no flags Details
Patches for master (2.68 KB, patch)
2014-12-12 15:18 UTC, Stefan Metzmacher
no flags Details
Cheery-pick for v4-2 of patches that went into master. (3.03 KB, patch)
2014-12-12 22:42 UTC, Jeremy Allison
metze: review+
Details
Patches for v4-1-test (4.29 KB, patch)
2014-12-14 11:08 UTC, Stefan Metzmacher
jra: review+
Details
Patches for v4-0-test (4.29 KB, patch)
2014-12-14 11:09 UTC, Stefan Metzmacher
jra: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Kyriacos Georgiou 2014-11-24 13:12:42 UTC
Hi,

I have spent weeks trying to figure out why the connection was being reset between my clients and the samba servers. I am noticing 'Server exit (NT_STATUS_CONNECTION_RESET)' messages in the logs which causes our software (loaded from network share) to break.

The problem seems to be in source/smbd/smb2_server.c. 

Here is the situation: When the initial connection is made and a new session is created, everything is fine. But after the initial session expires in 8 hours or so AND signing is required, then the server responds with a 'NT_STATUS_ACCESS_DENIED' message (which doesn't make any sense) and the connection is being reset. 

Note: I have forced the server to use protocol SMB2_02 in smb.conf

Here is the following block of code. The issue can be resolved by simply deleting the 'return access denied' line which forces the connection to drop in the 'if signing is required' section.


NTSTATUS smbd_smb2_request_dispatch(struct smbd_smb2_request *req)

            .. code ..

        if (req->do_encryption) {

            .. code ..

        } else if (flags & SMB2_HDR_FLAG_SIGNED) {

            .. code ..

        } else if (opcode == SMB2_OP_CANCEL) {

            .. code ..

        } else if (signing_required) {
                /*
                 * If signing is required we try to sign
                 * a possible error response
                 */
                req->do_signing = true;
                return smbd_smb2_request_error(req, NT_STATUS_ACCESS_DENIED);
        }


I hope I made this clear enough. There is something wrong with the logic here. The point is that the connection is to be kept alive and the client and server negotiate and create a new session.

Let me know if you have any queries.

Thanks,

Kyriacos
Comment 1 Jeremy Allison 2014-11-24 18:59:58 UTC
Sounds like kerberos ticket expiry time.

Can you get a wireshark trace of the connection around this time ? That will tell us why the client is sending an unsigned request on a signed connection.
Comment 2 Kyriacos Georgiou 2014-11-25 15:01:51 UTC
Created attachment 10460 [details]
wireshark packet capture

I have set the session expiry on samba to 1 minute for convenience and then tried replicating the issue.
Comment 3 Marcin Mogielnicki 2014-11-27 16:57:15 UTC
Jeremy,

Thank you very much for looking into it. Kyriacos asked me to do some data digging for him and here are my findings.

Root cause of this issue is samba forcibly signing and expecting signed SMB2_OP_SESSSETUP packets incoming. Is there specific reason for signing being enforced? No other opcodes are treated this way. I'd expect samba to follow smb.conf server signing setting on that.

Scenario we encounter is - client connects, as there is no signing key set up yet session nsinged packet is accepted and session is set up fine, signing key is created, signed response is sent to the client, no signing is used after that. Everything is up and running until server returns NT_STATUS_NETWORK_SESSION_EXPIRED. Client responds with unsigned session setup request, as expected. This time signing key is known to smbd, incoming packet is unsigned and for that reason smbd responds with signed NT_STATUS_ACCESS_DENIED straight away.

All findings are for samba 4.1.11 and SMB2_02 protocol.

Snippets of the code that demonstrate that session setup requests/responses are forcibly signed when signing_key is known:

For session request require incoming packet to be signed when signing key is known:

source3/smbd/smb2_server.c:
1910         if (x != NULL) {
1911                 signing_required = x->global->signing_required;
1912                 encryption_required = x->global->encryption_required;
1913
1914                 if (opcode == SMB2_OP_SESSSETUP &&
1915                     x->global->channels[0].signing_key.length) {
1916                         signing_required = true;
1917                 }
1918         }

and then, when session setup request is not signed, it reaches this bit of code:

2017         } else if (signing_required) {
2018                 /*
2019                  * If signing is required we try to sign
2020                  * a possible error response
2021                  */
2022                 req->do_signing = true;
2023                 return smbd_smb2_request_error(req, NT_STATUS_ACCESS_DENIED);
2024         }


Response is always signed no matter what settings are, this is what enforces that:

source3/smbd/smb2_sesssetup.c:
366         smb2req->session = session;
367         if (!guest) {
368                 smb2req->do_signing = true;
369         }
Comment 4 Stefan Metzmacher 2014-12-03 09:14:15 UTC
The capture shows that this happens with NTLMSSP where we use
GENSEC_EXPIRE_TIME_INFINITY.

So there's a another root cause for your specific problem.

Which Samba version are you using on which exact platform?
Comment 5 Marcin Mogielnicki 2014-12-03 15:13:44 UTC
Stefan,

That's 4.1.11 security ads clustered setup on Linux CentOS 6.2. Dev testbed for this issue is 4.1.11 ad member single machine (no ctdb) on SLES11SP1, this one is affected as well. In both cases max server protocol is set to SMB2_02. For diagnosing this issue the code was modified on dev setup to trigger session expiration after one minute, so I can tell you where expiry time is set up straight away:

source3/smbd/smb2_sesssetup.c
345         session->global->expiration_time = gensec_expire_time(session->gensec);

On production server (no modifications other than adding DEBUG call) shows that, when this line is reached right after smbd startup, session->gensec->name is spnego and session expiry time in session data is roughly eight hours in the future what matches expirty time of kerberos ticket ad is supposed to grant.

Is expiry time expected to be GENSEC_EXPIRE_TIME_INFINITY in ad member setup as well? Logically encountered behaviour makes sense - smbd is ad member, ad server issues kerberos ticket, that ticket needs renewing in eight hours, session expiry time reflects that.
Comment 6 Stefan Metzmacher 2014-12-05 14:04:04 UTC
(In reply to Marcin Mogielnicki from comment #5)

In the capture I see the NETWORK_SESSION_EXPIRED after 2 minutes
and there's no kerberos used at all...
Comment 7 Marcin Mogielnicki 2014-12-05 17:11:23 UTC
Stefan,

There is no kerberos traffic in client dump because samba server is the one who speaks to ad server, not the client.

Win 7 client sends session setup request, samba server gets one and speaks to ad server who issues kerberos ticket that needs renewing in eight hours, samba sets up session expiry time to eight hours. Client is unaware of any exchange of packets between samba server and ad server, you are not seeing one in included dump.

Let's rewind a little bit and focus on logic issues demonstrated in attached dump and being hardcoded in samba code - I pointed to related bits of code before. Security modes of neither client nor server enforce signing and yet:

1) packet no. 12, which is session setup response sent by samba server, is signed. Why?
2) packet no. 124, which is unsigned sesstion setup request, is rejected by samba server (I pointed to relevant bit of code before again) because one wants it signed. Why does one want it signed?
3) packet no. 125, which is "access denied" sesstion setup resonse, is signed. Why again?

Is there any spec that enforces that? I run quick comparison against widely known commercial vendor solution and one is not doing that, i.e. no packets are signed at all.
Comment 8 Stefan Metzmacher 2014-12-11 08:29:08 UTC
(In reply to Marcin Mogielnicki from comment #7)

I somehow didn't notice "For diagnosing this issue the code was modified on dev setup to trigger session expiration after one minute, so I can tell you where expiry time is set up straight away", sorry!

I thought the capture was created with an unmodified samba.
Comment 9 Stefan Metzmacher 2014-12-11 08:47:01 UTC
(In reply to Marcin Mogielnicki from comment #7)

> Win 7 client sends session setup request, samba server gets one and speaks to 
> ad server who issues kerberos ticket that needs renewing in eight hours, samba > sets up session expiry time to eight hours. Client is unaware of any exchange 
> of packets between samba server and ad server, you are not seeing one in 
> included dump.

The connection from winbindd to the domain controller is not relevant at all...

> 1) packet no. 12, which is session setup response sent by samba server, is 
> signed. Why?

This is what Windows also does and SMB 3.10 will require this in order to
detect downgrade attacks.

> 2) packet no. 124, which is unsigned sesstion setup request, is rejected by 
> samba server (I pointed to relevant bit of code before again) because one wants > it signed. Why does one want it signed?

This seems to be a bug.

> 3) packet no. 125, which is "access denied" sesstion setup resonse, is signed. > Why again?

This is just the consequence of 2).

In order to implement the correct fix I need to additional captures.

a) Please use the 1min hack and set "server signing = mandatory"
   on the server.

b) Please use the 1min hack and set "RequireSecuritySignature = 1"
   in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\LanmanWorkStation\Parameters on the client

I hope that the reauthentication will work fine in both cases
as the client uses signing. The fix would be to remove 

1914                 if (opcode == SMB2_OP_SESSSETUP &&
1915                     x->global->channels[0].signing_key.length) {
1916                         signing_required = true;
1917                 }

If that's not the case we need to ask Microsoft.
Comment 10 Kyriacos Georgiou 2014-12-11 10:46:27 UTC
Created attachment 10521 [details]
server signing mandatory
Comment 11 Kyriacos Georgiou 2014-12-11 10:47:04 UTC
Created attachment 10522 [details]
server signing mandatory and client signing required
Comment 12 Stefan Metzmacher 2014-12-11 11:03:28 UTC
(In reply to Kyriacos Georgiou from comment #11)

Can you also please upload client required only?
Comment 13 Stefan Metzmacher 2014-12-11 11:06:24 UTC
(In reply to Kyriacos Georgiou from comment #11)

There are no reauth attempts in any of these captures...
Comment 14 Kyriacos Georgiou 2014-12-11 11:17:00 UTC
When the server signing was set to mandatory, the connection couldn't even be established. As you can see from the two log files (server_signing..)


When I deleted " server signing mandatory" from the config, the client was able to establish connection. I am not capturing the data and waiting for expiry to occur. I will upload the file in a few minutes.

Thanks,

Kyriacos
Comment 15 Stefan Metzmacher 2014-12-11 11:29:07 UTC
(In reply to Kyriacos Georgiou from comment #14)

> When the server signing was set to mandatory, the connection couldn't even be 
> established. As you can see from the two log files (server_signing..)

Ok, the final session setup response is now not signed...
Did you modify this?

Can you please upload your full diff and tell me again which exact samba
version (commit) your're using.
Comment 16 Kyriacos Georgiou 2014-12-11 11:39:27 UTC
Created attachment 10523 [details]
packet capture after reg key set to enforce signing on client
Comment 17 Kyriacos Georgiou 2014-12-11 12:11:45 UTC
(In reply to Stefan (metze) Metzmacher from comment #15)

I think we got lost in all these messages, so let me make this clear.

Apart from changing the expiry time, nothing else was changed on samba source. The issues were discovered on official release Samba 4.1.13 and the testing happened on official release Samba 4.1.11. The behavior is the same on both versions.

------------------------------------------------------------------
Here are the results from all the tests performed up to this point.

1. Initial problem (NO signing enforced on either side): 
Connection is established, everything is fine until session expires. Then server demands signing and breaks connection because client doesn't do signing. Attachment:  wireshark packet capture


2. Test from today (Signing enforced on server using smb.conf variable):
Connection is NOT established at the beginning. Attachment: server signing mandatory


3. Test from today (Signing enforce on server using smb.conf variable AND enforced on client using reg key):
Again, connection NOT established at the beginning. Attachment: server signing mandatory and client signing required


4. Final test from today: (Signing enforced on client using reg key):
Success! Connection is established, everything is fine and when the session expires both sides agree and re-negotiate. Expected behaviour :-) Attachment:  packet capture after reg key set to enforce signing on client
------------------------------------------------------------------

I believe your fix will resolve the issue. I could apply your patch and try again if you want.

Thanks,

Kyriacos
Comment 18 Stefan Metzmacher 2014-12-12 15:18:31 UTC
Created attachment 10530 [details]
Patches for master
Comment 19 Kyriacos Georgiou 2014-12-12 17:12:04 UTC
(In reply to Stefan (metze) Metzmacher from comment #18)

Thanks Stefan!

Much appreciated...

What happens with the issue when signing was set to mandatory on the server though? (The session setup fails because more processing is required???)
Comment 20 Stefan Metzmacher 2014-12-12 20:51:21 UTC
(In reply to Kyriacos Georgiou from comment #19)

> What happens with the issue when signing was set to mandatory on the server 
> though? (The session setup fails because more processing is required???)

I can't reproduce this with a windows 2012r2 client.
Comment 21 Jeremy Allison 2014-12-12 22:42:58 UTC
Created attachment 10531 [details]
Cheery-pick for v4-2 of patches that went into master.
Comment 22 Stefan Metzmacher 2014-12-13 20:30:14 UTC
Comment on attachment 10531 [details]
Cheery-pick for v4-2 of patches that went into master.

We also need patches for v4-0-test and v4-1-test
Comment 23 Stefan Metzmacher 2014-12-14 11:08:51 UTC
Created attachment 10533 [details]
Patches for v4-1-test
Comment 24 Stefan Metzmacher 2014-12-14 11:09:37 UTC
Created attachment 10534 [details]
Patches for v4-0-test
Comment 25 Jeremy Allison 2014-12-15 18:56:28 UTC
Comment on attachment 10533 [details]
Patches for v4-1-test

LGTM.
Comment 26 Jeremy Allison 2014-12-15 19:11:30 UTC
Comment on attachment 10534 [details]
Patches for v4-0-test

LGTM.
Comment 27 Jeremy Allison 2014-12-15 19:15:24 UTC
Re-assigning to Karolin for inclusion in 4.2.0, 4.1.next, 4.0.next.
Comment 28 Karolin Seeger 2014-12-15 20:04:22 UTC
Pushed to autobuild-v4-[0|1|2]-test.
Comment 29 Stefan Metzmacher 2014-12-17 11:44:24 UTC
(In reply to Karolin Seeger from comment #28)

Pushed to v4-2-test.
Comment 30 Karolin Seeger 2014-12-20 15:52:17 UTC
(In reply to Stefan (metze) Metzmacher from comment #29)
Pushed to all branches.
Closing out bug report.

Thanks!