Hi, I have spent weeks trying to figure out why the connection was being reset between my clients and the samba servers. I am noticing 'Server exit (NT_STATUS_CONNECTION_RESET)' messages in the logs which causes our software (loaded from network share) to break. The problem seems to be in source/smbd/smb2_server.c. Here is the situation: When the initial connection is made and a new session is created, everything is fine. But after the initial session expires in 8 hours or so AND signing is required, then the server responds with a 'NT_STATUS_ACCESS_DENIED' message (which doesn't make any sense) and the connection is being reset. Note: I have forced the server to use protocol SMB2_02 in smb.conf Here is the following block of code. The issue can be resolved by simply deleting the 'return access denied' line which forces the connection to drop in the 'if signing is required' section. NTSTATUS smbd_smb2_request_dispatch(struct smbd_smb2_request *req) .. code .. if (req->do_encryption) { .. code .. } else if (flags & SMB2_HDR_FLAG_SIGNED) { .. code .. } else if (opcode == SMB2_OP_CANCEL) { .. code .. } else if (signing_required) { /* * If signing is required we try to sign * a possible error response */ req->do_signing = true; return smbd_smb2_request_error(req, NT_STATUS_ACCESS_DENIED); } I hope I made this clear enough. There is something wrong with the logic here. The point is that the connection is to be kept alive and the client and server negotiate and create a new session. Let me know if you have any queries. Thanks, Kyriacos
Sounds like kerberos ticket expiry time. Can you get a wireshark trace of the connection around this time ? That will tell us why the client is sending an unsigned request on a signed connection.
Created attachment 10460 [details] wireshark packet capture I have set the session expiry on samba to 1 minute for convenience and then tried replicating the issue.
Jeremy, Thank you very much for looking into it. Kyriacos asked me to do some data digging for him and here are my findings. Root cause of this issue is samba forcibly signing and expecting signed SMB2_OP_SESSSETUP packets incoming. Is there specific reason for signing being enforced? No other opcodes are treated this way. I'd expect samba to follow smb.conf server signing setting on that. Scenario we encounter is - client connects, as there is no signing key set up yet session nsinged packet is accepted and session is set up fine, signing key is created, signed response is sent to the client, no signing is used after that. Everything is up and running until server returns NT_STATUS_NETWORK_SESSION_EXPIRED. Client responds with unsigned session setup request, as expected. This time signing key is known to smbd, incoming packet is unsigned and for that reason smbd responds with signed NT_STATUS_ACCESS_DENIED straight away. All findings are for samba 4.1.11 and SMB2_02 protocol. Snippets of the code that demonstrate that session setup requests/responses are forcibly signed when signing_key is known: For session request require incoming packet to be signed when signing key is known: source3/smbd/smb2_server.c: 1910 if (x != NULL) { 1911 signing_required = x->global->signing_required; 1912 encryption_required = x->global->encryption_required; 1913 1914 if (opcode == SMB2_OP_SESSSETUP && 1915 x->global->channels[0].signing_key.length) { 1916 signing_required = true; 1917 } 1918 } and then, when session setup request is not signed, it reaches this bit of code: 2017 } else if (signing_required) { 2018 /* 2019 * If signing is required we try to sign 2020 * a possible error response 2021 */ 2022 req->do_signing = true; 2023 return smbd_smb2_request_error(req, NT_STATUS_ACCESS_DENIED); 2024 } Response is always signed no matter what settings are, this is what enforces that: source3/smbd/smb2_sesssetup.c: 366 smb2req->session = session; 367 if (!guest) { 368 smb2req->do_signing = true; 369 }
The capture shows that this happens with NTLMSSP where we use GENSEC_EXPIRE_TIME_INFINITY. So there's a another root cause for your specific problem. Which Samba version are you using on which exact platform?
Stefan, That's 4.1.11 security ads clustered setup on Linux CentOS 6.2. Dev testbed for this issue is 4.1.11 ad member single machine (no ctdb) on SLES11SP1, this one is affected as well. In both cases max server protocol is set to SMB2_02. For diagnosing this issue the code was modified on dev setup to trigger session expiration after one minute, so I can tell you where expiry time is set up straight away: source3/smbd/smb2_sesssetup.c 345 session->global->expiration_time = gensec_expire_time(session->gensec); On production server (no modifications other than adding DEBUG call) shows that, when this line is reached right after smbd startup, session->gensec->name is spnego and session expiry time in session data is roughly eight hours in the future what matches expirty time of kerberos ticket ad is supposed to grant. Is expiry time expected to be GENSEC_EXPIRE_TIME_INFINITY in ad member setup as well? Logically encountered behaviour makes sense - smbd is ad member, ad server issues kerberos ticket, that ticket needs renewing in eight hours, session expiry time reflects that.
(In reply to Marcin Mogielnicki from comment #5) In the capture I see the NETWORK_SESSION_EXPIRED after 2 minutes and there's no kerberos used at all...
Stefan, There is no kerberos traffic in client dump because samba server is the one who speaks to ad server, not the client. Win 7 client sends session setup request, samba server gets one and speaks to ad server who issues kerberos ticket that needs renewing in eight hours, samba sets up session expiry time to eight hours. Client is unaware of any exchange of packets between samba server and ad server, you are not seeing one in included dump. Let's rewind a little bit and focus on logic issues demonstrated in attached dump and being hardcoded in samba code - I pointed to related bits of code before. Security modes of neither client nor server enforce signing and yet: 1) packet no. 12, which is session setup response sent by samba server, is signed. Why? 2) packet no. 124, which is unsigned sesstion setup request, is rejected by samba server (I pointed to relevant bit of code before again) because one wants it signed. Why does one want it signed? 3) packet no. 125, which is "access denied" sesstion setup resonse, is signed. Why again? Is there any spec that enforces that? I run quick comparison against widely known commercial vendor solution and one is not doing that, i.e. no packets are signed at all.
(In reply to Marcin Mogielnicki from comment #7) I somehow didn't notice "For diagnosing this issue the code was modified on dev setup to trigger session expiration after one minute, so I can tell you where expiry time is set up straight away", sorry! I thought the capture was created with an unmodified samba.
(In reply to Marcin Mogielnicki from comment #7) > Win 7 client sends session setup request, samba server gets one and speaks to > ad server who issues kerberos ticket that needs renewing in eight hours, samba > sets up session expiry time to eight hours. Client is unaware of any exchange > of packets between samba server and ad server, you are not seeing one in > included dump. The connection from winbindd to the domain controller is not relevant at all... > 1) packet no. 12, which is session setup response sent by samba server, is > signed. Why? This is what Windows also does and SMB 3.10 will require this in order to detect downgrade attacks. > 2) packet no. 124, which is unsigned sesstion setup request, is rejected by > samba server (I pointed to relevant bit of code before again) because one wants > it signed. Why does one want it signed? This seems to be a bug. > 3) packet no. 125, which is "access denied" sesstion setup resonse, is signed. > Why again? This is just the consequence of 2). In order to implement the correct fix I need to additional captures. a) Please use the 1min hack and set "server signing = mandatory" on the server. b) Please use the 1min hack and set "RequireSecuritySignature = 1" in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\LanmanWorkStation\Parameters on the client I hope that the reauthentication will work fine in both cases as the client uses signing. The fix would be to remove 1914 if (opcode == SMB2_OP_SESSSETUP && 1915 x->global->channels[0].signing_key.length) { 1916 signing_required = true; 1917 } If that's not the case we need to ask Microsoft.
Created attachment 10521 [details] server signing mandatory
Created attachment 10522 [details] server signing mandatory and client signing required
(In reply to Kyriacos Georgiou from comment #11) Can you also please upload client required only?
(In reply to Kyriacos Georgiou from comment #11) There are no reauth attempts in any of these captures...
When the server signing was set to mandatory, the connection couldn't even be established. As you can see from the two log files (server_signing..) When I deleted " server signing mandatory" from the config, the client was able to establish connection. I am not capturing the data and waiting for expiry to occur. I will upload the file in a few minutes. Thanks, Kyriacos
(In reply to Kyriacos Georgiou from comment #14) > When the server signing was set to mandatory, the connection couldn't even be > established. As you can see from the two log files (server_signing..) Ok, the final session setup response is now not signed... Did you modify this? Can you please upload your full diff and tell me again which exact samba version (commit) your're using.
Created attachment 10523 [details] packet capture after reg key set to enforce signing on client
(In reply to Stefan (metze) Metzmacher from comment #15) I think we got lost in all these messages, so let me make this clear. Apart from changing the expiry time, nothing else was changed on samba source. The issues were discovered on official release Samba 4.1.13 and the testing happened on official release Samba 4.1.11. The behavior is the same on both versions. ------------------------------------------------------------------ Here are the results from all the tests performed up to this point. 1. Initial problem (NO signing enforced on either side): Connection is established, everything is fine until session expires. Then server demands signing and breaks connection because client doesn't do signing. Attachment: wireshark packet capture 2. Test from today (Signing enforced on server using smb.conf variable): Connection is NOT established at the beginning. Attachment: server signing mandatory 3. Test from today (Signing enforce on server using smb.conf variable AND enforced on client using reg key): Again, connection NOT established at the beginning. Attachment: server signing mandatory and client signing required 4. Final test from today: (Signing enforced on client using reg key): Success! Connection is established, everything is fine and when the session expires both sides agree and re-negotiate. Expected behaviour :-) Attachment: packet capture after reg key set to enforce signing on client ------------------------------------------------------------------ I believe your fix will resolve the issue. I could apply your patch and try again if you want. Thanks, Kyriacos
Created attachment 10530 [details] Patches for master
(In reply to Stefan (metze) Metzmacher from comment #18) Thanks Stefan! Much appreciated... What happens with the issue when signing was set to mandatory on the server though? (The session setup fails because more processing is required???)
(In reply to Kyriacos Georgiou from comment #19) > What happens with the issue when signing was set to mandatory on the server > though? (The session setup fails because more processing is required???) I can't reproduce this with a windows 2012r2 client.
Created attachment 10531 [details] Cheery-pick for v4-2 of patches that went into master.
Comment on attachment 10531 [details] Cheery-pick for v4-2 of patches that went into master. We also need patches for v4-0-test and v4-1-test
Created attachment 10533 [details] Patches for v4-1-test
Created attachment 10534 [details] Patches for v4-0-test
Comment on attachment 10533 [details] Patches for v4-1-test LGTM.
Comment on attachment 10534 [details] Patches for v4-0-test LGTM.
Re-assigning to Karolin for inclusion in 4.2.0, 4.1.next, 4.0.next.
Pushed to autobuild-v4-[0|1|2]-test.
(In reply to Karolin Seeger from comment #28) Pushed to v4-2-test.
(In reply to Stefan (metze) Metzmacher from comment #29) Pushed to all branches. Closing out bug report. Thanks!