Bug 15418 - secure channel faulty since Windows 10/11 update 07/2023
Summary: secure channel faulty since Windows 10/11 update 07/2023
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: AD: LDB/DSDB/SAMDB (show other bugs)
Version: 4.17.8
Hardware: All All
: P5 critical with 5 votes (vote)
Target Milestone: ---
Assignee: Jule Anger
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-07-12 14:54 UTC by Samuel Wolf
Modified: 2023-09-08 12:22 UTC (History)
20 users (show)

See Also:


Attachments
log.samba.bz2 (14.35 KB, application/x-bzip)
2023-07-13 10:09 UTC, Felix Botner
no flags Details
WIP patch for master (2.25 KB, patch)
2023-07-14 08:22 UTC, Ralph Böhme
no flags Details
Patches for v4-18-test (12.30 KB, patch)
2023-07-17 07:47 UTC, Stefan Metzmacher
abartlet: review+
metze: review? (slow)
Details
Patches for v4-17-test (12.30 KB, patch)
2023-07-17 07:48 UTC, Stefan Metzmacher
abartlet: review+
metze: review? (slow)
Details
Patches for v4-16-test (12.30 KB, patch)
2023-07-17 07:52 UTC, Stefan Metzmacher
abartlet: review+
metze: review? (slow)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Samuel Wolf 2023-07-12 14:54:00 UTC
Since we install the most recent windows updates from 07/2023
Test-ComputerSecureChannel -Verbose
show False [4].

Moved the machine to local workgroup, deleted machine account on the
ad controller and rejoined it (which works), but Test-ComputerSecureChannel -Verbose
show again False.

I'm not sure this is an issue on the samba or on the windows side.

Affected samba versions so far:
4.17.8+dfsg-2 (Debian 12) [1]
4.18.x (mjt packages) [2]
4.15.13+dfsg-0 (ubuntu) [3]

Thank you.

Samuel

[1] https://lists.samba.org/archive/samba/2023-July/245749.html
[2] https://lists.samba.org/archive/samba/2023-July/245763.html
[3] https://lists.samba.org/archive/samba/2023-July/245761.html

[4] https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/test-computersecurechannel?view=powershell-5.1

[5] https://lists.samba.org/archive/samba/2023-July/245765.html
Comment 1 Samuel Wolf 2023-07-12 14:54:37 UTC
What I see when I try to repair the secure channel
(Test-ComputerSecureChannel -Repair) with log level 5/6:

[...]
[2023/07/11 23:02:11.348659,  3]
../../libcli/auth/schannel_state_tdb.c:129(schannel_store_session_key_tdb)
  schannel_store_session_key_tdb: stored schannel info with key
SECRETS/SCHANNEL/CADMACHINE
[2023/07/11 23:02:11.348684,  1] ../../librpc/ndr/ndr.c:668(_ndr_push_error)
  ndr_push_netr_Capabilities: ndr_push_error(Bad Switch): Bad switch
value 2 at librpc/gen_ndr/ndr_netlogon.c:7652
[...]

But, I don't know this was the same output with an older windows patch
level, so maybe useless.
Comment 2 Samuel Wolf 2023-07-12 15:11:57 UTC
With windows updates less than 07/2023
Test-ComputerSecureChannel -Verbose
show True
Comment 3 Samuel Wolf 2023-07-12 16:10:31 UTC
More log (level 10):

[2023/07/12 18:04:47.063433, 10, pid=11202, effective(0, 0), real(0, 0)] ../../librpc/rpc/server/netlogon/schannel_util.c:187(dcesrv_netr_check_schannel_once)
  CVE-2020-1472(ZeroLogon)/CVE-2022-38023: netr_LogonGetCapabilities request (opnum[21]) WITH SEALED schannel from client_account[CADMACHINE$] client_computer_name[CADMACHINE] NT_STATUS_OK
[2023/07/12 18:04:47.063445,  1, pid=11202, effective(0, 0), real(0, 0), class=rpc_parse] ../../librpc/ndr/ndr.c:490(ndr_print_function_debug)
       netr_LogonGetCapabilities: struct netr_LogonGetCapabilities
          out: struct netr_LogonGetCapabilities
              return_authenticator     : *
                  return_authenticator: struct netr_Authenticator
                      cred: struct netr_Credential
                          data                     : XXXXXXXXXXXXXXXX
                      timestamp                : (time_t)0
              capabilities             : *
                  capabilities             : union netr_Capabilities(case 2)
                  UNKNOWN LEVEL 2
              result                   : NT_STATUS_NOT_SUPPORTED
[2023/07/12 18:04:47.063578,  1, pid=11202, effective(0, 0), real(0, 0), class=rpc_parse] ../../librpc/ndr/ndr.c:668(_ndr_push_error)
  ndr_push_netr_Capabilities: ndr_push_error(Bad Switch): Bad switch value 2 at librpc/gen_ndr/ndr_netlogon.c:7652
Comment 4 Ralph Böhme 2023-07-12 18:08:10 UTC
Hm, neither librpc/idl/netlogin.idl nor https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-nrpc/f03cad67-077f-4042-80b0-cdc38dca9968 have a level 2.
Comment 5 Jeremy Allison 2023-07-12 18:17:11 UTC
Looks like an urgent dochelp is needed. Microsoft are supposed to push updates to the docs before (or at least concurrent with) and code changes visible on the wire.
Comment 6 Michael Saxl 2023-07-12 18:23:41 UTC
very annoying. just debugged and after I saw Bad switch value 2 at librpc/gen_ndr/ndr_netlogon.c:7652 I immediately found this bug report.

Hope Microsoft takes back this update asap since it will take at least a half year until a fix for this will be downstream
Comment 7 Joerg Pulz 2023-07-13 06:27:11 UTC
The MS docs say this:

Verify that the QueryLevel parameter is set to 1. All other values are invalid, and STATUS_INVALID_LEVEL MUST be returned.

But ...

Looking at the code at function
_netr_LogonGetCapabilities() in source3/rpc_server/netlogon/srv_netlog_nt.c 
and
dcesrv_netr_LogonGetCapabilities() in source4/rpc_server/netlogon/dcerpc_netlogon.c

there is this:

if (r->in.query_level != 1) {
        return NT_STATUS_NOT_SUPPORTED;
}

So are we wrong here?
I changed this from NT_STATUS_NOT_SUPPORTED to NT_STATUS_INVALID_LEVEL here but it changed nothing.
The error still remains the same.

Just wondering if it should be changed in our code anyway, just to make sure that we comply with the specs.
Comment 8 Michael Saxl 2023-07-13 08:52:25 UTC
(In reply to Joerg Pulz from comment #7)
I think the issue is not the query_level but the "type" of the NETLOGON_CAPABILITIES

type 1 is a uint32 bitmask of type netr_NegotiateFlags.

looking at the bitmap they used 26 out of 32, so they have only 4 missing. Maybe they now have type 2 of type uint64?
Comment 9 Ralph Böhme 2023-07-13 09:11:26 UTC
(In reply to Jeremy Allison from comment #5)
https://lists.samba.org/archive/cifs-protocol/2023-July/004004.html
Comment 10 Michael Saxl 2023-07-13 09:23:57 UTC
just did a very lazy test and made a case(2) that holds the same bitmask (32bit)

the whole thing was 0. Don't know the exact byte ordering, but if the data on the wire is network byte order it could really be that they extended the bitmask.

Wonder however when they did introduce case 2 server side. Very old MS DC's might fail the same way samba is failing.
Comment 11 Ralph Böhme 2023-07-13 09:32:56 UTC
(In reply to Michael Saxl from comment #10)
Can you provide a log with level 10? Iirc at that level the log contains the raw bytes of the decrypted packets.
Comment 12 Felix Botner 2023-07-13 10:09:20 UTC
Created attachment 17979 [details]
log.samba.bz2

Hi,

here the log.samba (level 10) from a failed logon with an domain account on a windows 10 (with the KB5028166 update).

Best regard,
Felix
Comment 13 Ralph Böhme 2023-07-13 10:34:21 UTC
(In reply to Felix Botner from comment #12)
Thanks!

It seems the client's netr_LogonGetCapabilities request has a query-level of 2:

[2023/07/13 11:59:17.063300,  1, pid=32385, effective(0, 0), real(0, 0), class=rpc_parse] ../../librpc/ndr/ndr.c:484(ndr_print_function_debug)
       netr_LogonGetCapabilities: struct netr_LogonGetCapabilities
          in: struct netr_LogonGetCapabilities
              server_name              : *
                  server_name              : '\\master.five.new'
              computer_name            : *
                  computer_name            : 'DESKTOP-6O7C598'
              credential               : *
                  credential: struct netr_Authenticator
                      cred: struct netr_Credential
                          data                     : b43363f1a6823757
                      timestamp                : Thu Jul 13 11:59:16 2023 CEST
              return_authenticator     : *
                  return_authenticator: struct netr_Authenticator
                      cred: struct netr_Credential
                          data                     : 0000000000000000
                      timestamp                : (time_t)0
              query_level              : 0x00000002 (2)

According to https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-nrpc/5780fc6c-82f0-489f-b9a0-a9e855388492 the server should fail this with STATUS_INVALID_LEVEL.

Some accepts the request and then later uses the query-level as switch into the previously mentioned netr_Capabilities union. This then fails when trying to marshall the structure.

Read this with a grain of salt, I'm not our netlogon expert... :)
Comment 15 Stefan Metzmacher 2023-07-13 11:24:52 UTC
(In reply to Ralph Böhme from comment #14)

The patch in https://cpaste.org/?376fc231e8184be0#BcUKYAR2nbqWnZKn9py3Cb1y94mit4nFrcbWYjBTbJK2
is NOT a FIX!

It can be used in a temporary member server in a domain
with a latest windows DC. Log level = 100 should reveal
how a windows dc replies in the same situation...

We need the all winbindd related logs

Please note the resulting winbindd is not functional at all!
It's only there in order to collect information!

This is a more persistent link to the change:
https://cpaste.org/?da37e4398824e644#92j1V3XvRq8q46FUBGUvbJ4Vq5QS7pqyvpm7XfFbfz6N
Comment 16 Stefan Metzmacher 2023-07-13 11:30:50 UTC
(In reply to Stefan Metzmacher from comment #15)

The result against an unpatched windows dc would also be very useful!
Comment 17 Michael Saxl 2023-07-13 11:41:59 UTC
sorry for the noise today.

as you already guessed the previous things I wrote are totally wrong reversing in and out data.

I think what Samuel Wolf said is correct (we need to respond with NT_STATUS_INVALID_LEVEL), but I suspect that will not be returned since ndr_push_error seems to take precedence (in wireshark I see the last call fail with nca_s_fault_ndr)

given that the message

ndr_push_error(Bad Switch): Bad switch value 2 at librpc/gen_ndr/ndr_netlogon.c:7652

is after

result                   : NT_STATUS_NOT_SUPPORTED

I suspect that it is samba failing to send the response (with the NT_STATUS_INVALID_LEVEL)
Comment 19 Michael Saxl 2023-07-13 12:00:51 UTC
(In reply to Stefan Metzmacher from comment #15)
I have no idea how exactly rpc works in this case, but https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-nrpc/5780fc6c-82f0-489f-b9a0-a9e855388492
says that PNETLOGON_CAPABILITIES ServerCapabilities is a output, so it is something that samba dc responds (or in the client case the dc sends to the client).

The format (Level) is mandated by the client. Currently only 1 is documented.

In rpc speaking how should
[out, switch_is(QueryLevel)] PNETLOGON_CAPABILITIES ServerCapabilities
handle the case where QueryLevel is unknown but the return value of the function/rpc call is a failure?

In plain c the output pointer this would probably simply write nothing to the output, but how does rpc encode such things?

does
[default,flag(NDR_REMAINING)] DATA_BLOB _unknown;
actually send "nothing" (in the outgoing direction, inbound I could guess what it does)?
Comment 20 Stefan Metzmacher 2023-07-13 12:37:31 UTC
(In reply to Stefan Metzmacher from comment #16)

This could be a shortterm fix in order to behave like an unpatched windows server:

https://cpaste.org/?2cf886b026bb425b#3v2grejdN3GiAEqpzqjhd8gN1eMfqpoWngKz2REvSB2T

 source3/rpc_server/netlogon/srv_netlog_nt.c   | 1 +
 source4/rpc_server/netlogon/dcerpc_netlogon.c | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/source3/rpc_server/netlogon/srv_netlog_nt.c b/source3/rpc_server/netlogon/srv_netlog_nt.c
index 3ba58e61206f..9aea36a9d08e 100644
--- a/source3/rpc_server/netlogon/srv_netlog_nt.c
+++ b/source3/rpc_server/netlogon/srv_netlog_nt.c
@@ -2297,6 +2297,7 @@ NTSTATUS _netr_LogonGetCapabilities(struct pipes_struct *p,
 	}
 
 	if (r->in.query_level != 1) {
+		p->fault_state = DCERPC_NCA_S_FAULT_INVALID_TAG;
 		return NT_STATUS_NOT_SUPPORTED;
 	}
 
diff --git a/source4/rpc_server/netlogon/dcerpc_netlogon.c b/source4/rpc_server/netlogon/dcerpc_netlogon.c
index 6ccba65d3bf0..464093479966 100644
--- a/source4/rpc_server/netlogon/dcerpc_netlogon.c
+++ b/source4/rpc_server/netlogon/dcerpc_netlogon.c
@@ -2376,7 +2376,7 @@ static NTSTATUS dcesrv_netr_LogonGetCapabilities(struct dcesrv_call_state *dce_c
 	NT_STATUS_NOT_OK_RETURN(status);
 
 	if (r->in.query_level != 1) {
-		return NT_STATUS_NOT_SUPPORTED;
+		DCESRV_FAULT(DCERPC_NCA_S_FAULT_INVALID_TAG);
 	}
 
 	r->out.capabilities->server_capabilities = creds->negotiate_flags;
Comment 21 Stefan Metzmacher 2023-07-13 12:42:32 UTC
(In reply to Michael Saxl from comment #17)

Samba it actually generating DCERPC_FAULT_BAD_STUB_DATA,
while a windows server generated DCERPC_NCA_S_FAULT_INVALID_TAG.
Comment 22 Stefan Metzmacher 2023-07-13 12:43:41 UTC
(In reply to Michael Saxl from comment #19)

[default,flag(NDR_REMAINING)] DATA_BLOB _unknown; is actually wrong
the correct idl change will be:

 librpc/idl/netlogon.idl | 1 +
 1 file changed, 1 insertion(+)

diff --git a/librpc/idl/netlogon.idl b/librpc/idl/netlogon.idl
index c62920915b17..383c7b567e30 100644
--- a/librpc/idl/netlogon.idl
+++ b/librpc/idl/netlogon.idl
@@ -1236,6 +1236,7 @@ interface netlogon
 	/* Function 0x15 */
 	typedef [switch_type(uint32)] union {
 		[case(1)] netr_NegotiateFlags server_capabilities;
+		[case(2)] netr_NegotiateFlags server_capabilities;
 	} netr_Capabilities;
 
 	NTSTATUS netr_LogonGetCapabilities(
Comment 23 Michael Saxl 2023-07-13 12:58:45 UTC
(In reply to Stefan Metzmacher from comment #20)

just for my understanding:
does DCESRV_FAULT(DCERPC_NCA_S_FAULT_INVALID_TAG); actually send a rpc "protocol" level error whereas return NT_STATUS_NOT_SUPPORTED would set the return value of the rpc function?

according microsoft's doc I would read that the function returns INVALID, not the rpc layer itself. But thinking again that could really be the right way since how else would they be able to respond with a unknown union element
Comment 24 Michael Saxl 2023-07-13 13:36:40 UTC
(In reply to Stefan Metzmacher from comment #21)

tried that and can confirm that now we have a rpc call failt with nca_s_fault_invalid_tag

I don't see a output of the "out", so I guess the idl update is not needed in that case

after that I see NetrLogonGetDomainInfo request/response

That fails with credentials check failed (with the computer account)
(dcesrv_netr_LogonGetDomainInfo: Bad credentials - computer[name])
The result for the client is the same (NT_STATUS_TRUSTED_RELATIONSHIP_FAILURE)

I don't think I typed the password wrong (tried multiple times)
Also don't think the machine trust account has an issue, but I could try rejoining the machine if you think that could be the issue
Comment 25 Stefan Metzmacher 2023-07-13 14:06:08 UTC
(In reply to Michael Saxl from comment #24)

Thanks for testing, please try this:

https://cpaste.org/?df0494cac0063e2e#Cx69G684EBPQ71S6sAUVXSYburgV6gPyKHfPSbfmHZPJ

 source3/rpc_server/netlogon/srv_netlog_nt.c   | 9 +++++----
 source4/rpc_server/netlogon/dcerpc_netlogon.c | 8 ++++----
 2 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/source3/rpc_server/netlogon/srv_netlog_nt.c b/source3/rpc_server/netlogon/srv_netlog_nt.c
index 3ba58e61206f..2018dc28eb67 100644
--- a/source3/rpc_server/netlogon/srv_netlog_nt.c
+++ b/source3/rpc_server/netlogon/srv_netlog_nt.c
@@ -2284,6 +2284,11 @@ NTSTATUS _netr_LogonGetCapabilities(struct pipes_struct *p,
 	struct netlogon_creds_CredentialState *creds;
 	NTSTATUS status;
 
+	if (r->in.query_level != 1) {
+		p->fault_state = DCERPC_NCA_S_FAULT_INVALID_TAG;
+		return NT_STATUS_NOT_SUPPORTED;
+	}
+
 	become_root();
 	status = dcesrv_netr_creds_server_step_check(p->dce_call,
 						p->mem_ctx,
@@ -2296,10 +2301,6 @@ NTSTATUS _netr_LogonGetCapabilities(struct pipes_struct *p,
 		return status;
 	}
 
-	if (r->in.query_level != 1) {
-		return NT_STATUS_NOT_SUPPORTED;
-	}
-
 	r->out.capabilities->server_capabilities = creds->negotiate_flags;
 
 	return NT_STATUS_OK;
diff --git a/source4/rpc_server/netlogon/dcerpc_netlogon.c b/source4/rpc_server/netlogon/dcerpc_netlogon.c
index 6ccba65d3bf0..c869a6d3c791 100644
--- a/source4/rpc_server/netlogon/dcerpc_netlogon.c
+++ b/source4/rpc_server/netlogon/dcerpc_netlogon.c
@@ -2364,6 +2364,10 @@ static NTSTATUS dcesrv_netr_LogonGetCapabilities(struct dcesrv_call_state *dce_c
 	struct netlogon_creds_CredentialState *creds;
 	NTSTATUS status;
 
+	if (r->in.query_level != 1) {
+		DCESRV_FAULT(DCERPC_NCA_S_FAULT_INVALID_TAG);
+	}
+
 	status = dcesrv_netr_creds_server_step_check(dce_call,
 						     mem_ctx,
 						     r->in.computer_name,
@@ -2375,10 +2379,6 @@ static NTSTATUS dcesrv_netr_LogonGetCapabilities(struct dcesrv_call_state *dce_c
 	}
 	NT_STATUS_NOT_OK_RETURN(status);
 
-	if (r->in.query_level != 1) {
-		return NT_STATUS_NOT_SUPPORTED;
-	}
-
 	r->out.capabilities->server_capabilities = creds->negotiate_flags;
 
 	return NT_STATUS_OK;
Comment 26 Michael Saxl 2023-07-13 14:28:27 UTC
(In reply to Stefan Metzmacher from comment #25)

don't know if I have time to do that today.
But one thing I can already say: since responding with DCESRV_FAULT(DCERPC_NCA_S_FAULT_INVALID_TAG);
Windows tried (maybe only once, now it does not anymore, maybe a retry timer) with case 1 but then failed with the mentioned bad credentials.

In summary currenty "my" win11 now does not use case 2 anymore but fails in NetrLogonGetDomainInfo
Comment 27 Michael Saxl 2023-07-13 14:37:18 UTC
(In reply to Michael Saxl from comment #26)

rebooted pc, and can confirm it works!

Thanks Stefan Metzmacher

summarizing:

if the query_level is not 1 then we must not do anything and just do
DCESRV_FAULT(DCERPC_NCA_S_FAULT_INVALID_TAG);
(the extra call might, as you suggested, break things)

Love when things turn out easier than expected. This looks like a change that will not take ages to develop and backport

I think there is no need to change the idl file (currently I have the change in, but I don't see why that would be needed)
Comment 28 Michael Saxl 2023-07-13 15:12:34 UTC
Just for the sake of completeness:

I was a bit fooled by the huge debug log. What Windows acutally does is calling GetLogonCapabilities twice, the first time with query_level = 1, the second time with 2. if the second call failes with the way last patch does it continues like the second call never happened
Comment 29 Adi Kriegisch 2023-07-13 15:56:46 UTC
(In reply to Stefan Metzmacher from comment #25)

I can confirm this patch works on top of Debian's Samba (4.13.13+dfsg-1~deb11u5) in Bullseye. Thank you very much!
Comment 30 Björn Baumbach 2023-07-13 15:59:47 UTC
(In reply to Stefan Metzmacher from comment #25)
> https://cpaste.org/?df0494cac0063e2e#Cx69G684EBPQ71S6sAUVXSYburgV6gPyKHfPSbfmHZPJ

Works for me :-)

The pipeline was successful:
https://gitlab.com/samba-team/devel/samba/-/pipelines/930169521

The test with Windows 11 was successful.

Without Metze's patch:
root@laila:~# smbclient //192.168.4.100/share -UTEMP\\Administrator
session setup failed: NT_STATUS_TRUSTED_RELATIONSHIP_FAILURE

With Metze's patch:
root@laila:~# smbclient //192.168.4.100/share -UTEMP\\Administrator
Try "help" to get a list of possible commands.
smb: \>
Comment 31 Denis Cardon 2023-07-13 16:13:53 UTC
(In reply to Stefan Metzmacher from comment #25)
> https://cpaste.org/?df0494cac0063e2e#Cx69G684EBPQ71S6sAUVXSYburgV6gPyKHfPSbfmHZPJ

I confirm that the patch from Metze does fix the RDP connexion issues (tested on top of 4.18.4).

Thanks!

Denis
Comment 32 Carlos Cunha 2023-07-13 16:49:38 UTC
(In reply to Denis Cardon from comment #31)

Hi!

Sorry for my silly doubt, how to apply this patch in samba 4.18.4 Ubuntu via apt(deb http://www.corpit.ru/mjt/packages/samba focal/samba-4.18/).

regrads;
Comment 33 Samuel Wolf 2023-07-13 17:58:46 UTC
You can not patch without rebuild of samba, wait for new packages.

I hope Michael can build them for Debian 11/12 as well.
Comment 34 Michael Tokarev 2023-07-13 19:37:36 UTC
Answering here too, about the updated debian/ubuntu packages. Please wait for the actual samba patch to at least appear in the upstream git repository in the master branch.  It needs to complete all the CI tests before we can rush out and release packages for everything.
Comment 35 Ralph Böhme 2023-07-14 08:22:05 UTC
Created attachment 17983 [details]
WIP patch for master
Comment 36 Daniele Palumbo 2023-07-14 11:07:40 UTC
Do anyone has the reference of the downstream bugs opened?
Debian/Ubuntu/Fedora/RHEL/...
Comment 37 Michael Saxl 2023-07-14 11:28:19 UTC
ubuntu:
https://bugs.launchpad.net/ubuntu/+source/samba/+bug/2027716
Comment 38 Samuel Wolf 2023-07-14 11:45:48 UTC
@Michael Tokarev, do we need a bug report at Debian as well?
Comment 39 Krzysztof Olędzki 2023-07-14 12:01:30 UTC
Gentoo: https://bugs.gentoo.org/910306
Comment 40 Michael Tokarev 2023-07-14 12:20:33 UTC
Debian bug report: https://bugs.debian.org/1041043 . It's fixed in debian sid by 2:4.18.4+dfsg-2 upload today, update for bookworm has been uploaded too.
Comment 41 Krzysztof Olędzki 2023-07-14 12:21:06 UTC
I verified the patch on my test system running samba-4.17.8, thank you so much for a very quick fix!

That said, risking I'm stating the obvious thing, it seems to me that the patch is more a short-term workaround than the proper fix.

The protocol changes have been implemented to address a security problem, so while we are able to restore domain trust by properly rejecting request with query_level==2 and perhaps "abusing" the fall-back mechanism, longer term we need to properly support query_level==2. I also suspect that at some point, the fall-back mechanism in Windows client may stop working.

Next, if my reading of MS doc is correct, to properly address the security problem, not only we need to support query_level==2 but also reject request from clients with query_level==1, at least by default (perhaps a adding config entry, similar to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters).

On top of this, the client side also need to be implemented (unless this has been done already?) as otherwise Active Directory instances based on Windows will not allow Samba clients to connect. 

See: https://support.microsoft.com/en-us/topic/kb5021130-how-to-manage-the-netlogon-protocol-changes-related-to-cve-2022-38023-46ea3067-3989-4d40-963c-680fd9e8ee25#eventerrors5021130

Once again, sorry if I'm stating obvious things, and I really appropriate how quickly the initial patch was provided.
Comment 42 Michael Saxl 2023-07-14 12:28:47 UTC
(In reply to Krzysztof Olędzki from comment #41)
For that we would need doc from microsoft.

I don't think level 1 will go away since windows 10/11 does both calls (level 1 AND level 2, in that order. There is no "fallback")
Before the update it simply did not continue after the "wrong" failure.

This is also the reason why this issue will not happen with genuine ms dc (of any version)
Comment 43 Ralph Böhme 2023-07-14 13:06:33 UTC
(In reply to Krzysztof Olędzki from comment #41)
> Next, if my reading of MS doc is correct, to properly address the security problem,
> not only we need to support query_level==2 but also reject request from clients with 
> query_level==1, at least by default

where did you find this reference? Last time I looked online in MS-NRPC it wasn't reading that way, it only talked about level 1.
Comment 44 Krzysztof Olędzki 2023-07-14 13:21:32 UTC
(In reply to Ralph Böhme from comment #43)

Again, that's my reading of https://support.microsoft.com/en-us/topic/kb5021130-how-to-manage-the-netlogon-protocol-changes-related-to-cve-2022-38023-46ea3067-3989-4d40-963c-680fd9e8ee25, maybe incorrect.

So, this:
0 – Disabled
1 – Compatibility mode. Windows domain controllers will require that Netlogon clients use RPC Seal if they are running Windows, or if they are acting as either domain controllers or Trust accounts.
2 - Enforcement mode. All clients are required to use RPC Seal. See Change 2.


and "Timing of updates to address CVE-2022-38023":

* November 8, 2022 - Initial deployment phase:
Updates will be released in several phases: the initial phase for updates released on or after November 8, 2022 and the Enforcement phase for updates released on or after July 11, 2023.

* April 11, 2023 - Initial enforcement phase:
The initial deployment phase starts with the updates released on November 8, 2022 and continues with later Windows updates until the Enforcement phase. Windows updates on or after November 8, 2022 address security bypass vulnerability of CVE-2022-38023 by enforcing RPC sealing on all Windows clients.

By default, devices will be set in Compatibility mode. Windows domain controllers will require that Netlogon clients use RPC seal if they are running Windows, or if they are acting as either domain controllers or as trust accounts.

The Windows updates released on or after April 11, 2023 will remove the ability to disable RPC sealing by setting value 0 to the RequireSeal registry subkey.

* June 13, 2023 - Enforcement by Default:
The RequireSeal registry subkey will be moved to Enforced mode unless Administrators explicitly configure to be under Compatibility mode. Vulnerable connections from all clients including third-parties will be denied authentication. See Change 1.

* July 11, 2023 - Enforcement phase
The Windows updates released on July 11, 2023 will remove the ability to set value 1 to the RequireSeal registry subkey. This enables the Enforcement phase of CVE-2022-38023.
Comment 45 Michael Tokarev 2023-07-14 13:31:52 UTC
(In reply to Krzysztof Olędzki from comment #44)

The 0, 1, 2 values for the RequireSeal registry key are not about the message level (1 vs 2) which started this bug report.  It is about encrypting message exchange between two entities in a domain, not about the format of those messages.  It looks to me the two are not related, it just happens so that Microsoft has 2 changes in the same update, - one requiring Sealing and another introducing a new message format/level.

HTH
Comment 46 Ralph Böhme 2023-07-14 13:36:39 UTC
(In reply to Krzysztof Olędzki from comment #44)
This talks about whether to requiting sealed netlogon or not. Yes, it uses numbers 0, 1 and 2 for the configuration, but that's about it. :)

Samba support sealed (encrypted) netlogon since at least 4.0. The problem in this bugreport is about the query level in https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-nrpc/5780fc6c-82f0-489f-b9a0-a9e855388492 which also uses the numbers 1 and the undocumented 2, which is causing the problems.
Comment 47 Ralph Böhme 2023-07-14 13:37:22 UTC
(In reply to Michael Tokarev from comment #45)
very well summarized! :)
Comment 48 Robert Giles 2023-07-14 14:28:11 UTC
(In reply to Stefan Metzmacher from comment #25)

(I wanted to reply yesterday morning, but had to wait a day for Bugzilla account creation)

Confirming Stefan's workaround on 4.15.7, too;  I didn't even need to restart the client Windows 10 PC to restore trust.
Comment 49 Krzysztof Olędzki 2023-07-14 14:32:43 UTC
(In reply to Robert Giles from comment #48)

Yes, I also noticed that trust gets *eventually* restored even w/o a restart but it takes some time. Restart accelerates it.
Comment 50 Samuel Wolf 2023-07-14 18:00:53 UTC
I can't get it to work:

[2023/07/14 19:46:56.541130,  3] ../../libcli/auth/schannel_state_tdb.c:198(schannel_fetch_session_key_tdb)
  schannel_fetch_session_key_tdb: restored schannel info key SECRETS/SCHANNEL/CADMACHINE
[2023/07/14 19:46:56.541262,  5] ../../libcli/auth/credentials.c:257(netlogon_creds_step)
  	seed        XXXXXXXXXXXXXXXXX
[2023/07/14 19:46:56.541344,  5] ../../libcli/auth/credentials.c:263(netlogon_creds_step)
  	seed+time   XXXXXXXXXXXXXXXXX
[2023/07/14 19:46:56.541427,  5] ../../libcli/auth/credentials.c:272(netlogon_creds_step)
  	CLIENT      XXXXXXXXXXXXXXXXX
[2023/07/14 19:46:56.541531,  5] ../../libcli/auth/credentials.c:278(netlogon_creds_step)
  	seed+time+1 XXXXXXXXXXXXXXXXX
[2023/07/14 19:46:56.541619,  5] ../../libcli/auth/credentials.c:286(netlogon_creds_step)
  	SERVER      XXXXXXXXXXXXXXXXX
[2023/07/14 19:46:56.541714,  2] ../../libcli/auth/credentials.c:682(netlogon_creds_server_check_internal)
  credentials check failed
[2023/07/14 19:46:56.541831,  0] ../../source4/rpc_server/netlogon/dcerpc_netlogon.c:2623(dcesrv_netr_LogonGetDomainInfo)
  dcesrv_netr_LogonGetDomainInfo: Bad credentials - computer[CADMACHINE] remote[ipv4:192.168.44.20:49700] local[ipv4:192.168.44.60:49152]

Or is this any problem with the Debian Bookworm version samba_4.17.8+dfsg-2 I patched?
Comment 51 matt 2023-07-14 18:48:49 UTC
Just to add another data point, yesterday we used Stefan's patch on samba-4.10.16-24 on RHEL 7.9 for our ancient NT4 domain, and it fixed the problem.   The patch needed a small tweak to apply but it seems good.  Thank you.
Comment 52 Michael Saxl 2023-07-14 19:51:28 UTC
(In reply to Samuel Wolf from comment #50)
did you use the latest patch of metze?

the second last had that behavior. (the one immediatly before I announce the success should work, to be exact the one where the check is moved to the beginning of the function)
Comment 53 Michael Tokarev 2023-07-14 19:59:26 UTC
(In reply to Michael Saxl from comment #52)
to be extra clear: the one which is attached to this bug report in the "Attachments" section at the beginning, not the ones in the comments.
Comment 54 Samuel Wolf 2023-07-14 20:48:31 UTC
(In reply to Michael Saxl from comment #52)
> did you use the latest patch of metze?

You are right, I use (for whatever reason) unintentionally the wrong patch.
Thanks for the hint!
Comment 55 Samuel Wolf 2023-07-15 09:02:52 UTC
I can confirm that the patch works!
Thanks to everyone for searching, patching and testing.
Comment 56 Michael Lueck 2023-07-15 12:49:11 UTC
(In reply to Krzysztof Olędzki from comment #44)


I would rather get Windows 10 Pro clients reconnecting to our Samba NT4 domain without applying a binary patch to our trusty / stable Samba configuration. I would be all right applying an update to the smb.conf / just not patching binaries please.

So I tried the:

RequireSeal subkey
Registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters
Value: RequireSeal
Data type: REG_DWORD
Data: 0 – Disabled

suggestion. I inspected one Windows 10 Pro registry prior to updating it. The Parameters key did NOT already have the RequireSeal value.

Restarting the Windows 10 Pro client, again I get the C-A-D screen error that "The Trust Relationship between this Workstation and the Domain failed" type message. So I have to unplug the workstation from the LAN, login to it with cached domain credentials, then pop the LAN cable back in.

I see other similar named values in that same Key...

REGEDIT4

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters]
"DisablePasswordChange"=dword:00000000
"MaximumPasswordAge"=dword:0000001e
"RequireSignOrSeal"=dword:00000001
"RequireStrongKey"=dword:00000001
"SealSecureChannel"=dword:00000001
"ServiceDll"=hex(2):25,53,79,73,74,65,6d,52,6f,6f,74,25,5c,73,79,73,74,65,6d,\
  33,32,5c,6e,65,74,6c,6f,67,6f,6e,2e,64,6c,6c,00
"SignSecureChannel"=dword:00000001
"Update"="no"
"RequireSeal"=dword:00000000



Perhaps do some of the other "Requires*" values also need to change to 0 to revert Microsoft's impact?
Comment 57 Michael Tokarev 2023-07-15 13:12:18 UTC
(In reply to Michael Lueck from comment #56)
As has been noted before a few times, starting with Jul-2023 windows update, RequireSeal parameter is ignored and the sealing is always required instead.
Comment 58 Michael Tokarev 2023-07-15 15:08:24 UTC
https://lists.debian.org/debian-stable-announce/2023/07/msg00000.html is the official debian release as well.
Comment 59 Michael Lueck 2023-07-15 15:38:36 UTC
(In reply to Michael Tokarev from comment #57)
> starting with Jul-2023 windows update, RequireSeal parameter is ignored and the sealing is always required instead.


So this means it is now a mandatory server update?

If so, rats! I am happily running on back-level Ubuntu server. No more updates to the Samba packages.
Comment 60 Samba QA Contact 2023-07-17 07:36:12 UTC
This bug was referenced in samba master:

5f87888ed53320538cf773d64868390d8641a40e
404ce08e9088968311c714e756f5d58ce2cef715
d5f1097b6220676d56ed5fc6707acf667b704518
dfeabce44fbb78083fbbb2aa634fc4172cf83db9
Comment 61 Stefan Metzmacher 2023-07-17 07:47:25 UTC
Created attachment 17985 [details]
Patches for v4-18-test
Comment 62 Stefan Metzmacher 2023-07-17 07:48:16 UTC
Created attachment 17986 [details]
Patches for v4-17-test
Comment 63 Stefan Metzmacher 2023-07-17 07:52:24 UTC
Created attachment 17987 [details]
Patches for v4-16-test
Comment 64 Stefan Metzmacher 2023-07-17 08:02:15 UTC
Comment on attachment 17987 [details]
Patches for v4-16-test

I guess this is important enough to be included in 4.16
Comment 65 Andrew Bartlett 2023-07-17 08:07:33 UTC
Comment on attachment 17987 [details]
Patches for v4-16-test

This is absolutely a reasonable this for the 4.16 series, given it un-breaks a security release made on the MS side.
Comment 66 Andrew Bartlett 2023-07-17 08:08:28 UTC
Assigning to Jule for all maintained branches.
Comment 67 Stefano B. 2023-07-18 07:25:08 UTC
(In reply to Stefan Metzmacher from comment #25)


I confirm that the patch by Metzmacher also works on Oracle Linux Server release 6.10 and Samba 4.8.1. 

Thank you

Stefano B.
Comment 68 Stefano B. 2023-07-18 07:25:50 UTC
(In reply to Stefan Metzmacher from comment #25)


I confirm that the patch by Metzmacher also works on Oracle Linux Server release 6.10 and Samba 4.8.1. 

Thank you

Stefano B.
Comment 69 bunkobugsy 2023-07-18 11:57:33 UTC
Thank you, patch also works fine adapted for 4.10.16-24 on RHEL7 in PDC mode.

Curios, we've had intermittent errors with exactly the same message months before MS released this July patch, but they went away after a restart and we couldn't really track down the cause.

Is there any chance that query_level != 1 is treated the same wrong way somewhere else in samba code?
I'm not sure we can reproduce error and capture level 10 logs for a debug.
Comment 70 Piviul 2023-07-19 06:27:41 UTC
Hi all, the patch works for my win7/win10 clients but for my olds windows 2000/xp doesn't :(

I'm the only one that still has old win2k/xp clients? I know are very old unsupported distributions but I can't still free my LAN from 2k/xp clients...

:(

Piviul
Comment 71 Michael Tokarev 2023-07-19 08:10:53 UTC
(In reply to Piviul from comment #70)
This whole issue has exactly nothing to do with old (xp/2k) clients. As the title says, it is about current win10/11 *updated* this month.  This change does not alter samba in any way when old clients are connected.  And the support for the old protocols which must be not used today will be dropped in future samba versions, just like it is dropped already in current windows.  If you need old unsupported clients, you have to use old unsupported samba (and probably old unsupported linux too).  Please don't post irrelevant-to-the topic questions.
Comment 72 Stefan Metzmacher 2023-07-19 08:28:54 UTC
Can someone please upload a level 10 log file of against a working/patched Samba server? That would allow us to implement a similar logic in our client.
Comment 73 Samba QA Contact 2023-07-19 14:29:20 UTC
This bug was referenced in samba v4-16-stable (Release samba-4.16.11):

6e87858185f33318e6206ca831d46643172beb47
5d48ea23decc42ba6f28bd8589ab27630cbe1670
585df7e0220b57c022b0a7218e52ec811ee5b05e
caf21883fa83a3277a1df405c32eeeda3950495b
Comment 74 Samba QA Contact 2023-07-19 14:30:56 UTC
This bug was referenced in samba v4-17-stable (Release samba-4.17.10):

492a52b1c4c97667d711efe1410aace18e940cf0
e14a5c36123ac01c91851cb40483e6251d9d43e9
55d0a38601236b89871f1a2f2bf7ad36c590f1f4
56fad90eaef07d11665c35ffc872f34165496076
Comment 75 Samba QA Contact 2023-07-19 14:31:58 UTC
This bug was referenced in samba v4-18-stable (Release samba-4.18.5):

0d7376cac6921e07a2c0722c64ec467c48ef1166
30c3fa71a17ce2ef3fde4325d4fc24556c3023e5
7818b00d13df1816363d513e9d5ecc7147b53792
be141a1c14b41c55e47db4b7265890afec2c0c09
Comment 76 Samba QA Contact 2023-07-19 14:56:59 UTC
This bug was referenced in samba v4-16-test:

6e87858185f33318e6206ca831d46643172beb47
5d48ea23decc42ba6f28bd8589ab27630cbe1670
585df7e0220b57c022b0a7218e52ec811ee5b05e
caf21883fa83a3277a1df405c32eeeda3950495b
Comment 77 Samba QA Contact 2023-07-19 15:00:26 UTC
This bug was referenced in samba v4-17-test:

492a52b1c4c97667d711efe1410aace18e940cf0
e14a5c36123ac01c91851cb40483e6251d9d43e9
55d0a38601236b89871f1a2f2bf7ad36c590f1f4
56fad90eaef07d11665c35ffc872f34165496076
Comment 78 Samba QA Contact 2023-07-19 15:07:42 UTC
This bug was referenced in samba v4-18-test:

0d7376cac6921e07a2c0722c64ec467c48ef1166
30c3fa71a17ce2ef3fde4325d4fc24556c3023e5
7818b00d13df1816363d513e9d5ecc7147b53792
be141a1c14b41c55e47db4b7265890afec2c0c09
Comment 79 John Michael Drescher 2023-07-19 16:53:27 UTC
(In reply to matt from comment #51)
Thanks for this info. Last week on my ancient NT4 domain I ended up going to every client (dozens) and uninstalling the KB5028166 patch and blocking it for now. I will try the patch.
Comment 80 Michael Saxl 2023-07-20 05:42:54 UTC
The devs probably already know that, but on the launchpad bugtracker Douglas Bagnall mentioned the updated doc:

https://winprotocoldoc.blob.core.windows.net/productionwindowsarchives/MS-NRPC/%5bMS-NRPC%5d-20230718-diff.pdf

The case(2) field is actually called RequestedFlags

ServerCapabilities: A 32-bit set of bit flags that identify the server's capabilities (section 3.5.4.4.10).
RequestedFlags: A 32-bit set of bit flags that identify the client capabilities that server received during negotiation (section 3.5.4.4.10).

That probably explains why returning both times the server capabilities did not work in my lazy tests :)
Comment 81 Michael Saxl 2023-07-20 06:51:48 UTC
just tried to make query_level reply with the flags I got in dcesrv_netr_ServerAuthenticate3 (I extended the internal creds structure with requested_flags and copied the flags over in dcesrv_netr_ServerAuthenticate3_helper)

now it works even with query level 2 (I reply requested_flags in the case 2)

The flags look the same, but the hex number is different, so there seem to be new flags

creds: struct netlogon_creds_CredentialState
        negotiate_flags          : 0x612fffff (1630535679)
               1: NETLOGON_NEG_ACCOUNT_LOCKOUT
               1: NETLOGON_NEG_PERSISTENT_SAMREPL
               1: NETLOGON_NEG_ARCFOUR     
               1: NETLOGON_NEG_PROMOTION_COUNT
               1: NETLOGON_NEG_CHANGELOG_BDC
               1: NETLOGON_NEG_FULL_SYNC_REPL
               1: NETLOGON_NEG_MULTIPLE_SIDS
               1: NETLOGON_NEG_REDO
               1: NETLOGON_NEG_PASSWORD_CHANGE_REFUSAL
               1: NETLOGON_NEG_SEND_PASSWORD_INFO_PDC
               1: NETLOGON_NEG_GENERIC_PASSTHROUGH
               1: NETLOGON_NEG_CONCURRENT_RPC
               1: NETLOGON_NEG_AVOID_ACCOUNT_DB_REPL
               1: NETLOGON_NEG_AVOID_SECURITYAUTH_DB_REPL
               1: NETLOGON_NEG_STRONG_KEYS
               1: NETLOGON_NEG_TRANSITIVE_TRUSTS
               1: NETLOGON_NEG_DNS_DOMAIN_TRUSTS
               1: NETLOGON_NEG_PASSWORD_SET2
               1: NETLOGON_NEG_GETDOMAININFO
               1: NETLOGON_NEG_CROSS_FOREST_TRUSTS
               0: NETLOGON_NEG_NEUTRALIZE_NT4_EMULATION
               1: NETLOGON_NEG_RODC_PASSTHROUGH
               0: NETLOGON_NEG_SUPPORTS_AES_SHA2
               1: NETLOGON_NEG_SUPPORTS_AES
               1: NETLOGON_NEG_AUTHENTICATED_RPC_LSASS
               1: NETLOGON_NEG_AUTHENTICATED_RPC
        requested_flags          : 0xe12fffff (3778019327)
               1: NETLOGON_NEG_ACCOUNT_LOCKOUT
               1: NETLOGON_NEG_PERSISTENT_SAMREPL
               1: NETLOGON_NEG_ARCFOUR
               1: NETLOGON_NEG_PROMOTION_COUNT
               1: NETLOGON_NEG_CHANGELOG_BDC
               1: NETLOGON_NEG_FULL_SYNC_REPL
               1: NETLOGON_NEG_MULTIPLE_SIDS
               1: NETLOGON_NEG_REDO
               1: NETLOGON_NEG_PASSWORD_CHANGE_REFUSAL
               1: NETLOGON_NEG_SEND_PASSWORD_INFO_PDC
               1: NETLOGON_NEG_GENERIC_PASSTHROUGH
               1: NETLOGON_NEG_CONCURRENT_RPC
               1: NETLOGON_NEG_AVOID_ACCOUNT_DB_REPL
               1: NETLOGON_NEG_AVOID_SECURITYAUTH_DB_REPL
               1: NETLOGON_NEG_STRONG_KEYS
               1: NETLOGON_NEG_TRANSITIVE_TRUSTS
               1: NETLOGON_NEG_DNS_DOMAIN_TRUSTS
               1: NETLOGON_NEG_PASSWORD_SET2
               1: NETLOGON_NEG_GETDOMAININFO
               1: NETLOGON_NEG_CROSS_FOREST_TRUSTS
               0: NETLOGON_NEG_NEUTRALIZE_NT4_EMULATION
               1: NETLOGON_NEG_RODC_PASSTHROUGH
               0: NETLOGON_NEG_SUPPORTS_AES_SHA2
               1: NETLOGON_NEG_SUPPORTS_AES
               1: NETLOGON_NEG_AUTHENTICATED_RPC_LSASS
               1: NETLOGON_NEG_AUTHENTICATED_RPC
Comment 82 Stefan Metzmacher 2023-07-20 07:23:20 UTC
(In reply to Michael Saxl from comment #81)

I also have patches for this, see https://bugzilla.samba.org/show_bug.cgi?id=15425

Which client is sending 0xe12fffff ?
Comment 83 Stefan Metzmacher 2023-07-20 07:25:38 UTC
Fixes are released in samba-4.16.11, samba-4.17.10 and samba-4.18.5,
the follow up will be tracked in https://bugzilla.samba.org/show_bug.cgi?id=15425
Comment 84 Michael Saxl 2023-07-20 07:25:58 UTC
windows 11 22h2
Comment 85 Piviul 2023-07-20 14:43:15 UTC
(In reply to Michael Tokarev from comment #71)
Hi Michael I'm saying that after the patch installation on the DC windows 2000 and xp clients can't authenticate domain users any more. Your xp/2k clients continue to authenticate users even after the patch installation on the DC?

Piviul
Comment 86 Michael Saxl 2023-07-20 14:56:57 UTC
(In reply to Piviul from comment #85)
this gets a bit off topic, but as some already said this patch/issue has nothing to do with new security requirements. This patch has no way to break old clients since this only changes the way samba handles a unknown call (that will not happen with old clients)

that being said at least windows xp works with a patched 4.17 samba server (if configured that it is allowed to talk with it, that is enable NT1/SMB1 and adding
  server reject md5 schannel:<computername>$ = no
for every legacy host)

If you made the update coming from a unmaintained version this is what (intentionally for security reasons) broke it