Bug 14851 - Do not upper case FQDN hostname based target principal in s4 librpc code
Summary: Do not upper case FQDN hostname based target principal in s4 librpc code
Status: ASSIGNED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: DCE-RPCs and pipes (show other bugs)
Version: 4.15.0
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Stefan Metzmacher
QA Contact: Samba QA Contact
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-10-01 11:01 UTC by Alexander Bokovoy
Modified: 2022-01-19 06:26 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Bokovoy 2021-10-01 11:01:39 UTC
As found while testing dcerpcd changes in MR#1948, https://gitlab.com/samba-team/samba/-/merge_requests/1948#note_621616533, there is a behavior difference to MS-RPCE section 2.1.1.1 in s4's librpc code:

-----------------------------------------------------------------------------

I am using Samba Python bindings to connect to LSA pipe. Below is a fragment:
```
# -*- coding: utf-8 -*-
from samba import param
from samba import credentials
from samba.dcerpc import lsa

def get_lp(realm):
    lp = param.LoadParm()
    # lp.load_default()
    lp.set('realm',realm)
    lp.set('log level','100')
    lp.set('workgroup', 'ipa')
    lp.set('netbios name', 'dc')
    return lp

lp = get_lp("ipa.test")
creds = credentials.Credentials()
creds.set_username('admin')
creds.set_password('Secret123')
creds.set_kerberos_state(credentials.MUST_USE_KERBEROS)
creds.guess(lp)
creds.set_workstation("dc")

clsa = lsa.lsarpc('ncacn_ip_tcp:dc.ipa.test[sign,seal,krb5]', lp, creds)
objectAttribute = lsa.ObjectAttribute()
objectAttribute.sec_qos = lsa.QosInfo()
policy_handle = clsa.OpenPolicy2("", objectAttribute,
                                 (lsa.LSA_POLICY_VIEW_LOCAL_INFORMATION |
                                  lsa.LSA_POLICY_TRUST_ADMIN |
                                  lsa.LSA_POLICY_CREATE_SECRET)
                                )

```

Running it with `KRB5_TRACE=/dev/stderr python creds.py` shows that Samba RPC client code wants to use upper case FQDN hostname-based principal for the target service (`host/DC.IPA.TEST@IPA.TEST`), due to the code in `source4/librpc/rpc/dcerpc_sock.c:continue_socket_connect()` which unconditionally capitalizes the server name. This is unrelated to this MR but it would be good to clarify this without a need to require uppercased keys in server keytabs -- we never needed this in FreeIPA when running RPC services behind smbd.

According to MS-RPCE 2.1.1.1,

> When extensions that are not specified in sections 2.1.1 through 2.1.2 are enabled over the TCP transport protocol, the network address MUST be an IPv4 or IPv6 address or a server name.<2> The server name MUST be a Unicode string that represents either a NetBIOS host name (see [MS-NBTE] section 2.2.1) or a fully qualified domain name (see [RFC1035] section 3.1 and [RFC2181] section 11).

where NetBIOS name is case-sensitive and FQDN name is insensitive, so uppercasing them in all cases looks incorrect.

When I forced the target principal with `target_principal=cifs/dc.ipa.test@IPA.TEST` in the binding string, the auth passed and I was able to reach the LSA end-point authenticated.

-----------------------------------------------------------------------------

I think librpc should not upper case FQDN hostname-based principal for the target service.
Comment 1 Alexander Bokovoy 2021-10-01 11:03:06 UTC
Metze, I think this is a bug and needs fixed. I didn't get any comment on this on MR#1948 so I am filing a bug now as after merging it we'll see a regression in FreeIPA due to how librpc changes the hostname in the principal.
Comment 2 Jeremy Allison 2021-10-01 17:30:07 UTC
Alexander, how do we know when to capitalize the name here - i.e. when do we know it's a NetBIOS name that needs uppercasing ? Is it as simple as looking for a '.' in the name ?
Comment 3 Alexander Bokovoy 2021-10-01 17:45:38 UTC
My opinion is that we should not normalize the name at all. NetBIOS names are case sensitive and strictly speaking should not be capitalized.
See MS-NBTE 2.2.1: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-nbte/6f06fa0e-1dc4-4c41-accb-355aaf20546d

-----
This document clarifies the ambiguity by specifying that the name space is defined as sixteen 8-bit binary bytes, with no restrictions, except that the name SHOULD NOT<3> start with an asterisk (*).

Neither [RFC1001] nor [RFC1002] discusses whether names are case-sensitive. This document clarifies this ambiguity by specifying that because the name space is defined as sixteen 8-bit binary bytes, a comparison MUST be done for equality against the entire 16 bytes. As a result, NetBIOS names are inherently case-sensitive.
-----
Comment 4 Jeremy Allison 2021-10-01 17:52:22 UTC
Or do we just need to talloc_strdup() here and make sure the correct canonicalization is done in the callers of continue_socket_connect() (which IMHO would seem to be the correct thing to do) ?
Comment 5 Alexander Bokovoy 2021-10-01 17:54:25 UTC
I think that could be a reasonable fix. I would still suggest to not normalize the NetBIOS names — we've had customers with very weird names in past, even though it might be not a problem anymore.
Comment 6 Jeremy Allison 2021-10-01 17:57:06 UTC
(In reply to Alexander Bokovoy from comment #3)

So the correct fix is to just change this to talloc_strdup(), and expect the callers to have done any name canonicalization if they want it.

Sounds good - what would a regression test for this look like ?

(Ignoring https://gitlab.com/samba-team/samba/-/merge_requests/1948 as it isn't strictly related to it but this needs to be fixed before it can go in).
Comment 7 Alexander Bokovoy 2021-10-01 18:15:46 UTC
Regression test would be something similar to what I have in the description:

 - have a DC environment
 - enroll a client with mixed-case NetBIOS name
 - try to connect to it over ncacn_ip_tcp with Kerberos ticket

Looking into testprogs/blackbox/test_net_ads.sh, it already uses low-case NetBIOS name there for the enrolled client (sha1sum gives us low-cased output). However, in those tests we use s3 code, not s4 librpc so a test could be simply to extend that one to use s4's python bindings.
Comment 8 Jeremy Allison 2021-10-01 18:29:04 UTC
OK, I might need some help on that ("simply to extend that one to use s4's python bindings" :-). The "simply" part isn't obvious to me :-).

Can we make a test that fails first (mark as knownfail) and then removes it after the code change currently under test at:

https://gitlab.com/samba-team/devel/samba/-/pipelines/381100689

passes ? (Presuming it does pass, of course :-).
Comment 9 Jeremy Allison 2021-10-01 18:32:34 UTC
(In reply to Alexander Bokovoy from comment #7)

Also, correct me if I'm wrong but testprogs/blackbox/test_net_ads.sh is creating the machine accounts in AD, but isn't spinning up a member server to connect to.

So the "try to connect to it" part won't work unless we actually have a running member I think.
Comment 10 Alexander Bokovoy 2021-10-01 18:32:47 UTC
Looking at other tests, it supposed to have worked already:

$ git grep -A1 torture_suite_add_machine_workstation_rpc_iface_tcase
source4/torture/ntp/ntp_signd.c:        tcase = torture_suite_add_machine_workstation_rpc_iface_tcase(suite,
source4/torture/ntp/ntp_signd.c-                                  "signd", &ndr_table_netlogon, TEST_MACHINE_NAME);
--
source4/torture/rpc/netlogon.c: tcase = torture_suite_add_machine_workstation_rpc_iface_tcase(suite, "wkst",
source4/torture/rpc/netlogon.c-                                           &ndr_table_netlogon, TEST_MACHINE_NAME);
--
source4/torture/rpc/remote_pac.c:       tcase = torture_suite_add_machine_workstation_rpc_iface_tcase(suite, "netr-mem-arcfour",
source4/torture/rpc/remote_pac.c-                                                                     &ndr_table_netlogon, TEST_MACHINE_NAME_WKSTA);
--
source4/torture/rpc/remote_pac.c:       tcase = torture_suite_add_machine_workstation_rpc_iface_tcase(suite, "netr-mem-aes",
source4/torture/rpc/remote_pac.c-                                                                     &ndr_table_netlogon, TEST_MACHINE_NAME_WKSTA);
--
source4/torture/rpc/remote_pac.c:       tcase = torture_suite_add_machine_workstation_rpc_iface_tcase(suite, "netr-mem-arcfour",
source4/torture/rpc/remote_pac.c-                                                                     &ndr_table_netlogon, TEST_MACHINE_NAME_S4U2SELF_WKSTA);
--
source4/torture/rpc/remote_pac.c:       tcase = torture_suite_add_machine_workstation_rpc_iface_tcase(suite, "netr-mem-aes",
source4/torture/rpc/remote_pac.c-                                                                     &ndr_table_netlogon, TEST_MACHINE_NAME_S4U2SELF_WKSTA);
--
source4/torture/rpc/remote_pac.c:       tcase = torture_suite_add_machine_workstation_rpc_iface_tcase(suite, "netr-mem-arcfour",
source4/torture/rpc/remote_pac.c-                                                                     &ndr_table_netlogon, TEST_MACHINE_NAME_S4U2PROXY_WKSTA);
--
source4/torture/rpc/remote_pac.c:       tcase = torture_suite_add_machine_workstation_rpc_iface_tcase(suite, "netr-mem-aes",
source4/torture/rpc/remote_pac.c-                                                                     &ndr_table_netlogon, TEST_MACHINE_NAME_S4U2PROXY_WKSTA);
--
source4/torture/rpc/rpc.c:_PUBLIC_ struct torture_rpc_tcase *torture_suite_add_machine_workstation_rpc_iface_tcase(
source4/torture/rpc/rpc.c-                              struct torture_suite *suite,
--
source4/torture/rpc/samr_accessmask.c:  tcase = torture_suite_add_machine_workstation_rpc_iface_tcase(suite, "samr",
source4/torture/rpc/samr_accessmask.c-                                                                &ndr_table_samr,
--
source4/torture/rpc/spoolss_access.c:   rpc_tcase = torture_suite_add_machine_workstation_rpc_iface_tcase(suite, "workstation",
source4/torture/rpc/spoolss_access.c-                                                                     &ndr_table_spoolss,
--
source4/torture/rpc/torture_rpc.h:struct torture_rpc_tcase *torture_suite_add_machine_workstation_rpc_iface_tcase(
source4/torture/rpc/torture_rpc.h-                              struct torture_suite *suite, 

$ git grep '#define TEST_MACHINE_NAME'
source4/torture/ntp/ntp_signd.c:#define TEST_MACHINE_NAME "ntpsigndtest"
source4/torture/rpc/drsuapi.c:#define TEST_MACHINE_NAME "torturetest"
source4/torture/rpc/drsuapi_w2k8.c:#define TEST_MACHINE_NAME "torturetest"
source4/torture/rpc/forest_trust.c:#define TEST_MACHINE_NAME "lsatestmach"
source4/torture/rpc/netlogon.c:#define TEST_MACHINE_NAME "torturetest"
source4/torture/rpc/netlogon_crypto.c:#define TEST_MACHINE_NAME "torturetest"
source4/torture/rpc/remote_pac.c:#define TEST_MACHINE_NAME_BDC "torturepacbdc"
source4/torture/rpc/remote_pac.c:#define TEST_MACHINE_NAME_WKSTA "torturepacwksta"
source4/torture/rpc/remote_pac.c:#define TEST_MACHINE_NAME_S4U2SELF_BDC "tests4u2selfbdc"
source4/torture/rpc/remote_pac.c:#define TEST_MACHINE_NAME_S4U2SELF_WKSTA "tests4u2selfwk"
source4/torture/rpc/remote_pac.c:#define TEST_MACHINE_NAME_S4U2PROXY_WKSTA "tests4u2proxywk"
source4/torture/rpc/samlogon.c:#define TEST_MACHINE_NAME "samlogontest"
source4/torture/rpc/samsync.c:#define TEST_MACHINE_NAME "samsynctest"
source4/torture/rpc/schannel.c:#define TEST_MACHINE_NAME "schannel"

all those names are low-cased, however, the problem is not visible because ads_keytab_create_default() creates keytab entries with upper-cased name as well so they practically make the regression invisible.

I think we can actually have a test that filters out upper-cased name from a machine keytab and force use of the specific keytab in smbd configuration. Then uppercased version will not be present and rpc code will fail without a fix.
Comment 11 Jeremy Allison 2021-10-01 18:35:18 UTC
Woohoo ! That's the kind of insight I didn't have :-). Any chance I can impose on you to write this test ? I will certainly learn lots by reviewing it :-).
Comment 12 Alexander Bokovoy 2021-10-01 18:37:45 UTC
It is getting late here and my weekend is already packed (joys of post-house move cleanup/setup, heh) so I will try to find time next week for this test. At worst, I'll try to create something similar to IPA smbd setup as I need that anyway (without ipasam).
Comment 13 Jeremy Allison 2021-10-01 18:44:19 UTC
Ah, the ci run here:

https://gitlab.com/samba-team/devel/samba/-/pipelines/381100689

fails spectacularly with the minor 'talloc_strupper() -> talloc_strdup()' change :-(.

I think there may be more to the name canonicalization in source4 that we're seeing here at first glance (i.e. I'm guessing it's not being done right in a bunch of places).
Comment 14 Jeremy Allison 2021-10-01 19:25:31 UTC
Ignore the previous comment :-). Turns out talloc_strupper() and talloc_strdup() behave differently on being passed a NULL pointer :-).

Looks like this is the raw fix:

diff --git a/source4/librpc/rpc/dcerpc_sock.c b/source4/librpc/rpc/dcerpc_sock.c
index e7ecca73e3c..c1f1ee4664c 100644
--- a/source4/librpc/rpc/dcerpc_sock.c
+++ b/source4/librpc/rpc/dcerpc_sock.c
@@ -94,7 +94,14 @@ static void continue_socket_connect(struct composite_context *ctx)
        conn->srv_max_recv_frag = 5840;
 
        conn->transport.pending_reads = 0;
-       conn->server_name   = strupper_talloc(conn, s->target_hostname);
+       if (s->target_hostname != NULL) {
+               conn->server_name = talloc_strdup(conn, s->target_hostname);
+               if (conn->server_name == NULL) {
+                       close(sock_fd);
+                       composite_error(c, NT_STATUS_NO_MEMORY);
+                       return;
+               }
+       }
 
        rc = tstream_bsd_existing_socket(conn, sock_fd,
                                         &conn->transport.stream);
Comment 15 Jeremy Allison 2021-10-01 19:50:02 UTC
Yep, that was it. Passing ci now with the NULL check added :-).
Comment 17 Jeremy Allison 2021-10-01 21:18:44 UTC
Yep, now passes ci ! We just need the regression test and we'll be good to go I think (unless Metze can see something I can't).
Comment 18 Stefan Metzmacher 2021-10-03 13:18:25 UTC
(In reply to Alexander Bokovoy from comment #0)

servicePrincipalName values are case insensitive and a kerberos acceptor
should cope with any value or I'm I missing something?
Comment 19 Stefan Metzmacher 2021-10-03 13:21:03 UTC
(In reply to Jeremy Allison from comment #14)

> -       conn->server_name   = strupper_talloc(conn, s->target_hostname);
> +       if (s->target_hostname != NULL) {
> +               conn->server_name = talloc_strdup(conn, s->target_hostname);
> +               if (conn->server_name == NULL) {
> +                       close(sock_fd);
> +                       composite_error(c, NT_STATUS_NO_MEMORY);
> +                       return;
> +               }
> +       }

I don't against such a change, but I don't understand why this would
any difference to the problem Alexander described...
Comment 20 Alexander Bokovoy 2021-10-03 15:34:46 UTC
I am not sure why but this broke when I tested with the code in MR#1948.

FreeIPA itself does support case-insensitive TGS request and it works on Fedora 33 setup I have at home but breaks with MR#1948 test environment I had.

I'll do additional testing tomorrow and report about it.
Comment 21 Jeremy Allison 2021-10-03 23:35:58 UTC
(In reply to Stefan Metzmacher from comment #19)

I think this fix is the right thing to do - in no other place do we modify the case of the server name so it does seem a little odd to me that we arbitrarily uppercase it here.

It doesn't make a difference in our ci-tests but I really would like to fully understand it before pushing anything though :-).
Comment 22 Jeremy Allison 2021-10-07 15:50:26 UTC
Ping Alexander. Can you give us an update on any investigations of this ? (I'm too busy is of course a valid update :-).
Comment 23 Alexander Bokovoy 2021-10-07 15:56:07 UTC
Yes, sadly that was my case. I hoped to get it done on Monday and now it is end of Thursday...

I'll try to carve some time on Friday. Sorry for the delay...
Comment 24 Jeremy Allison 2021-10-07 16:40:24 UTC
No rush Alexander, Ralph wants metze to look over the code in https://gitlab.com/samba-team/samba/-/merge_requests/1948 before we merge and he's busy for a week or so anyway. If you don't get to it until next week there's no harm.
Comment 25 Jeremy Allison 2021-10-29 22:00:32 UTC
Ping ! Just want to let you know I haven't forgotten about this one :-).

We'll need a fix in the next 2 weeks or so once Metze evaluates the dcerpcd changes (soon...) :-).
Comment 26 Alexander Bokovoy 2021-10-30 06:16:25 UTC
I haven't forgotten either but been busy last two weeks with somewhat more urgent work on a set of other bugzillas (together with Metze and others). Hope to get my part finalized soon too.
Comment 27 Jeremy Allison 2022-01-13 22:45:44 UTC
Hi Alexander, just wanted to check in and see if this is still a problem now the dcerpcd code has gone in ?
Comment 28 Alexander Bokovoy 2022-01-18 15:34:01 UTC
Hi Jeremy,

I am currently trying to understand whether there is still a problem. ;)

I've got FreeIPA deployed against Samba git master build and things work without changes but I see strange access denials when accessing LSA pipe. The same code works in 4.15.

I need to debug it more but the core of the issue is below. It is log.rpcd_lsad content for Kerberos authenticated lsa connection using the script from this bug description. This, basically, the same Python code that we have in samba-tool domain join and in FreeIPA. What it does is LSA RPC connection using ncacn_ip_tcp and asks LSA OpenPolicy2. While we are authenticated, the policy request gets rejected as ACCESS_DENIED.

[2022/01/18 12:51:42,  3, pid=30166, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:2871(lp_do_section)
  Processing section "[global]"
  doing parameter workgroup = IPA
  doing parameter netbios name = F35
  doing parameter realm = IPA.TEST
  doing parameter kerberos method = dedicated keytab
  doing parameter dedicated keytab file = /etc/samba/samba.keytab
  doing parameter create krb5 conf = no
  doing parameter server role = IPA PRIMARY DOMAIN CONTROLLER
  doing parameter security = user
  doing parameter domain master = yes
  doing parameter max log size = 100000
  doing parameter log file = /var/log/samba/log.%m
  doing parameter passdb backend = ipasam:ldapi://%2fvar%2frun%2fslapd-IPA-TEST.socket
  doing parameter disable spoolss = yes
  doing parameter ldapsam:trusted = yes
  doing parameter ldap ssl = off
  doing parameter ldap suffix = dc=ipa,dc=test
  doing parameter ldap user suffix = cn=users,cn=accounts
  doing parameter ldap group suffix = cn=groups,cn=accounts
  doing parameter ldap machine suffix = cn=computers,cn=accounts
  doing parameter idmap config * : backend = tdb
  doing parameter idmap config * : range = 0 - 0
  doing parameter idmap config IPA : backend = sss
  doing parameter idmap config IPA : range = 1389000000 - 1389200000
  doing parameter max smbd processes = 1000
  doing parameter log level = 10
  doing parameter rpc start on demand helpers = false
[2022/01/18 12:51:42,  4, pid=30166, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:4011(lp_load_ex)
  pm_process() returned Yes
[2022/01/18 12:51:42,  7, pid=30166, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:4346(lp_servicenumber)
  lp_servicenumber: couldn't find homes
[2022/01/18 12:51:42,  8, pid=30166, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:1510(add_a_service)
  add_a_service: Creating snum = 0 for IPC$
[2022/01/18 12:51:42, 10, pid=30166, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:1552(hash_a_service)
  hash_a_service: creating servicehash
[2022/01/18 12:51:42, 10, pid=30166, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:1560(hash_a_service)
  hash_a_service: hashing index 0 for service name IPC$
[2022/01/18 12:51:42,  3, pid=30166, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:1672(lp_add_ipc)
  adding IPC service
[2022/01/18 12:51:42,  5, pid=30166, effective(0, 0), real(0, 0)] ../../source3/auth/auth_generic.c:250(auth3_generate_session_info_pac)
  ../../source3/auth/auth_generic.c:250OK: user: admin domain: IPA client: 192.168.122.19
[2022/01/18 12:51:42,  4, pid=30166, effective(0, 0), real(0, 0), class=auth_audit] ../../auth/auth_log.c:740(log_successful_authz_event_human_readable)
  Successful AuthZ: [DCE/RPC,krb5] user [IPA]\[admin] [S-1-5-21-3703471042-164549623-3970024037-500] at [Tue, 18 Jan 2022 12:51:42.080414 UTC] Remote host [ipv4:192.168.122.19:54756] local host [ipv4:192.168.122.19:49152]
  {"timestamp": "2022-01-18T12:51:42.080512+0000", "type": "Authorization", "Authorization": {"version": {"major": 1, "minor": 1}, "localAddress": "ipv4:192.168.122.19:49152", "remoteAddress": "ipv4:192.168.122.19:54756", "serviceDescription": "DCE/RPC",
 "authType": "krb5", "domain": "IPA", "account": "admin", "sid": "S-1-5-21-3703471042-164549623-3970024037-500", "sessionId": "d21b30f3-4a41-4a75-b2fa-08beb7ceb108", "logonServer": "F35", "transportProtection": "SEAL", "accountFlags": "0x00000010"}}
[2022/01/18 12:51:42, 10, pid=30166, effective(0, 0), real(0, 0)] ../../librpc/rpc/dcerpc_util.c:399(dcerpc_pull_auth_trailer)
  dcerpc_pull_auth_trailer: auth_pad_length 12
[2022/01/18 12:51:42, 10, pid=30166, effective(0, 0), real(0, 0), class=auth] ../../auth/kerberos/gssapi_helper.c:305(gssapi_unseal_packet)
  Unsealed 128 bytes, with 76 bytes header/signature.
[2022/01/18 12:51:42,  4, pid=30166, effective(0, 0), real(0, 0)] ../../source3/smbd/sec_ctx.c:206(push_sec_ctx)
  push_sec_ctx(0, 0) : sec_ctx_stack_ndx = 1
[2022/01/18 12:51:42,  4, pid=30166, effective(0, 0), real(0, 0)] ../../source3/smbd/sec_ctx.c:317(set_sec_ctx_internal)
  setting sec ctx (1389000000, 1389000000) - sec_ctx_stack_ndx = 1
[2022/01/18 12:51:42,  5, pid=30166, effective(0, 0), real(0, 0)] ../../libcli/security/security_token.c:51(security_token_debug)
  Security token SIDs (9):
    SID[  0]: S-1-5-21-3703471042-164549623-3970024037-500
    SID[  1]: S-1-5-21-3703471042-164549623-3970024037-512
    SID[  2]: S-1-18-1
    SID[  3]: S-1-1-0
    SID[  4]: S-1-5-2
    SID[  5]: S-1-5-11
    SID[  6]: S-1-5-32-544
    SID[  7]: S-1-22-1-1389000000
    SID[  8]: S-1-22-2-1389000000
   Privileges (0x        1FFFFFF0):
    Privilege[  0]: SeMachineAccountPrivilege
    Privilege[  1]: SeTakeOwnershipPrivilege
    Privilege[  2]: SeBackupPrivilege
    Privilege[  3]: SeRestorePrivilege
    Privilege[  4]: SeRemoteShutdownPrivilege
    Privilege[  5]: SePrintOperatorPrivilege
    Privilege[  6]: SeAddUsersPrivilege
    Privilege[  7]: SeDiskOperatorPrivilege
    Privilege[  8]: SeSecurityPrivilege
    Privilege[  9]: SeSystemtimePrivilege
    Privilege[ 10]: SeShutdownPrivilege
    Privilege[ 11]: SeDebugPrivilege
    Privilege[ 12]: SeSystemEnvironmentPrivilege
    Privilege[ 13]: SeSystemProfilePrivilege
    Privilege[ 14]: SeProfileSingleProcessPrivilege
    Privilege[ 15]: SeIncreaseBasePriorityPrivilege
    Privilege[ 16]: SeLoadDriverPrivilege
    Privilege[ 17]: SeCreatePagefilePrivilege
    Privilege[ 18]: SeIncreaseQuotaPrivilege
    Privilege[ 19]: SeChangeNotifyPrivilege
    Privilege[ 20]: SeUndockPrivilege
    Privilege[ 21]: SeManageVolumePrivilege
    Privilege[ 22]: SeImpersonatePrivilege
    Privilege[ 23]: SeCreateGlobalPrivilege
    Privilege[ 24]: SeEnableDelegationPrivilege
   Rights (0x               0):
[2022/01/18 12:51:42,  5, pid=30166, effective(0, 0), real(0, 0)] ../../source3/auth/token_util.c:873(debug_unix_user_token)
  UNIX token of user 1389000000
  Primary group is 1389000000 and contains 1 supplementary groups
  Group[  0]: 1389000000
[2022/01/18 12:51:42,  5, pid=30166, effective(1389000000, 1389000000), real(1389000000, 0)] ../../source3/smbd/uid.c:522(smbd_become_authenticated_pipe_user)
  Impersonated user: uid=(1389000000,1389000000), gid=(0,1389000000)
[2022/01/18 12:51:42,  1, pid=30166, effective(1389000000, 1389000000), real(1389000000, 0), class=rpc_parse] ../../librpc/ndr/ndr.c:484(ndr_print_function_debug)
       lsa_OpenPolicy2: struct lsa_OpenPolicy2
          in: struct lsa_OpenPolicy2
              system_name              : *
                  system_name              : ''
              attr                     : *
                  attr: struct lsa_ObjectAttribute
                      len                      : 0x00000000 (0)
                      root_dir                 : NULL
                      object_name              : NULL
                      attributes               : 0x00000000 (0)
                      sec_desc                 : NULL
                      sec_qos                  : *
                          sec_qos: struct lsa_QosInfo
                              len                      : 0x00000000 (0)
                              impersonation_level      : 0x0000 (0)
                              context_mode             : 0x00 (0)
                              effective_only           : 0x00 (0)
              access_mask              : 0x00000029 (41)
                     1: LSA_POLICY_VIEW_LOCAL_INFORMATION
                     0: LSA_POLICY_VIEW_AUDIT_INFORMATION
                     0: LSA_POLICY_GET_PRIVATE_INFORMATION
                     1: LSA_POLICY_TRUST_ADMIN   
                     0: LSA_POLICY_CREATE_ACCOUNT
                     1: LSA_POLICY_CREATE_SECRET 
                     0: LSA_POLICY_CREATE_PRIVILEGE
                     0: LSA_POLICY_SET_DEFAULT_QUOTA_LIMITS
                     0: LSA_POLICY_SET_AUDIT_REQUIREMENTS
                     0: LSA_POLICY_AUDIT_LOG_ADMIN
                     0: LSA_POLICY_SERVER_ADMIN  
                     0: LSA_POLICY_LOOKUP_NAMES  
                     0: LSA_POLICY_NOTIFICATION  
[2022/01/18 12:51:42,  4, pid=30166, effective(0, 0), real(0, 0)] ../../source3/smbd/sec_ctx.c:443(pop_sec_ctx)
  pop_sec_ctx (0, 0) - sec_ctx_stack_ndx = 0
[2022/01/18 12:51:42,  5, pid=30166, effective(0, 0), real(0, 0), class=rpc_srv] ../../librpc/rpc/dcesrv_core.c:1941(dcesrv_request)
  dcerpc fault in call lsarpc:2c - DCERPC_FAULT_ACCESS_DENIED
[2022/01/18 12:51:42,  3, pid=30166, effective(0, 0), real(0, 0), class=rpc_srv] ../../source3/rpc_server/rpc_server.c:257(ncacn_terminate_connection)
  ncacn_terminate_connection: Terminating connection - 'dcesrv: NT_STATUS_CONNECTION_DISCONNECTED'
[2022/01/18 12:51:42,  1, pid=30166, effective(0, 0), real(0, 0), class=rpc_parse] ../../librpc/ndr/ndr.c:435(ndr_print_debug)
       &worker->status: struct rpc_worker_status
          server_index             : 0x01 (1)
          worker_index             : 0x00 (0)
          num_clients              : 0x00000000 (0)
[2022/01/18 12:51:42, 10, pid=30166, effective(0, 0), real(0, 0)] ../../lib/messaging/messages_dgm.c:1463(messaging_dgm_send)
  messaging_dgm_send: Sending message to 30146
[2022/01/18 12:52:42, 10, pid=30166, effective(0, 0), real(0, 0)] ../../source3/lib/messages.c:421(messaging_recv_cb)
  messaging_recv_cb: Received message 0xd len 0 (num_fds:0) from 30146
[2022/01/18 12:52:42,  5, pid=30166, effective(0, 0), real(0, 0), class=rpc_srv] ../../librpc/rpc/dcesrv_core.c:2559(dcesrv_shutdown_ep_server)
  dcesrv_shutdown_ep_server: Shutting down DCE/RPC endpoint server 'lsarpc'
[2022/01/18 12:52:42,  5, pid=30166, effective(0, 0), real(0, 0), class=rpc_srv] ../../librpc/rpc/dcesrv_core.c:2559(dcesrv_shutdown_ep_server)
  dcesrv_shutdown_ep_server: Shutting down DCE/RPC endpoint server 'samr'
[2022/01/18 12:52:42,  5, pid=30166, effective(0, 0), real(0, 0), class=rpc_srv] ../../librpc/rpc/dcesrv_core.c:2559(dcesrv_shutdown_ep_server)
  dcesrv_shutdown_ep_server: Shutting down DCE/RPC endpoint server 'dssetup'
[2022/01/18 12:52:42,  5, pid=30166, effective(0, 0), real(0, 0), class=rpc_srv] ../../librpc/rpc/dcesrv_core.c:2559(dcesrv_shutdown_ep_server)
  dcesrv_shutdown_ep_server: Shutting down DCE/RPC endpoint server 'netlogon'

The code in _lsa_OpenPolicy2() didn't change much. I suspect we stumble on the same block that checks the transport we have since 2012. Somehow, this works in 4.15, as I said, and doesn't work here:

e1951d75fc49 source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2001-02-26 19:31:07 +0000  391) 
872f9655587a source3/rpc_server/srv_lsa_nt.c     (Andreas Schneider       2010-07-28 09:48:42 +0200  392) NTSTATUS _lsa_OpenPolicy2(struct pipes_struct *p,
6a77f8d2d933 source3/rpc_server/srv_lsa_nt.c     (Günther Deschner        2008-02-04 21:00:38 +0100  393)                         struct lsa_OpenPolicy2 *r)
e1951d75fc49 source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2001-02-26 19:31:07 +0000  394) {
8379d8cd532b source3/rpc_server/lsa/srv_lsa_nt.c (Volker Lendecke         2021-10-04 13:40:02 +0200  395)       struct dcesrv_call_state *dce_call = p->dce_call;
8379d8cd532b source3/rpc_server/lsa/srv_lsa_nt.c (Volker Lendecke         2021-10-04 13:40:02 +0200  396)       struct auth_session_info *session_info =
8379d8cd532b source3/rpc_server/lsa/srv_lsa_nt.c (Volker Lendecke         2021-10-04 13:40:02 +0200  397)               dcesrv_call_session_info(dce_call);
7f6bb48bdf23 source3/rpc_server/srv_lsa_nt.c     (Günther Deschner        2010-05-18 10:29:34 +0200  398)       struct security_descriptor *psd = NULL;
6e7648650528 source3/rpc_server/srv_lsa_nt.c     (Jean-François Micouleau 2001-12-17 23:03:23 +0000  399)       size_t sd_size;
0dde2106bb9a source3/rpc_server/lsa/srv_lsa_nt.c (Richard Sharpe          2015-05-14 18:08:27 -0700  400)       uint32_t des_access = r->in.access_mask;
0dde2106bb9a source3/rpc_server/lsa/srv_lsa_nt.c (Richard Sharpe          2015-05-14 18:08:27 -0700  401)       uint32_t acc_granted;
6e7648650528 source3/rpc_server/srv_lsa_nt.c     (Jean-François Micouleau 2001-12-17 23:03:23 +0000  402)       NTSTATUS status;
6e7648650528 source3/rpc_server/srv_lsa_nt.c     (Jean-François Micouleau 2001-12-17 23:03:23 +0000  403) 
bbf70e793c7b source3/rpc_server/lsa/srv_lsa_nt.c (Andreas Schneider       2012-06-25 18:45:35 +0200  404)       if (p->transport != NCACN_NP && p->transport != NCALRPC) {
bbf70e793c7b source3/rpc_server/lsa/srv_lsa_nt.c (Andreas Schneider       2012-06-25 18:45:35 +0200  405)               p->fault_state = DCERPC_FAULT_ACCESS_DENIED;
bbf70e793c7b source3/rpc_server/lsa/srv_lsa_nt.c (Andreas Schneider       2012-06-25 18:45:35 +0200  406)               return NT_STATUS_ACCESS_DENIED;
bbf70e793c7b source3/rpc_server/lsa/srv_lsa_nt.c (Andreas Schneider       2012-06-25 18:45:35 +0200  407)       }
bbf70e793c7b source3/rpc_server/lsa/srv_lsa_nt.c (Andreas Schneider       2012-06-25 18:45:35 +0200  408) 
459dc8f39c08 source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2009-05-18 15:44:03 -0700  409)       /* Work out max allowed. */
8379d8cd532b source3/rpc_server/lsa/srv_lsa_nt.c (Volker Lendecke         2021-10-04 13:40:02 +0200  410)       map_max_allowed_access(session_info->security_token,
8379d8cd532b source3/rpc_server/lsa/srv_lsa_nt.c (Volker Lendecke         2021-10-04 13:40:02 +0200  411)                              session_info->unix_token,
f7ff6bd1425c source3/rpc_server/srv_lsa_nt.c     (Günther Deschner        2009-06-29 20:34:03 +0200  412)                              &des_access);
6e7648650528 source3/rpc_server/srv_lsa_nt.c     (Jean-François Micouleau 2001-12-17 23:03:23 +0000  413) 
6e7648650528 source3/rpc_server/srv_lsa_nt.c     (Jean-François Micouleau 2001-12-17 23:03:23 +0000  414)       /* map the generic bits to the lsa policy ones */
d649a46078e6 source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2009-05-20 11:52:11 -0700  415)       se_map_generic(&des_access, &lsa_policy_mapping);
6e7648650528 source3/rpc_server/srv_lsa_nt.c     (Jean-François Micouleau 2001-12-17 23:03:23 +0000  416) 
6e7648650528 source3/rpc_server/srv_lsa_nt.c     (Jean-François Micouleau 2001-12-17 23:03:23 +0000  417)       /* get the generic lsa policy SD until we store it */
d649a46078e6 source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2009-05-20 11:52:11 -0700  418)       status = make_lsa_object_sd(p->mem_ctx, &psd, &sd_size, &lsa_policy_mapping,
d649a46078e6 source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2009-05-20 11:52:11 -0700  419)                       NULL, 0);
d649a46078e6 source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2009-05-20 11:52:11 -0700  420)       if (!NT_STATUS_IS_OK(status)) {
d649a46078e6 source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2009-05-20 11:52:11 -0700  421)               return status;
d649a46078e6 source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2009-05-20 11:52:11 -0700  422)       }
6e7648650528 source3/rpc_server/srv_lsa_nt.c     (Jean-François Micouleau 2001-12-17 23:03:23 +0000  423) 
8379d8cd532b source3/rpc_server/lsa/srv_lsa_nt.c (Volker Lendecke         2021-10-04 13:40:02 +0200  424)       status = access_check_object(psd, session_info->security_token,
62e5900cd13f source3/rpc_server/srv_lsa_nt.c     (Andrew Bartlett         2010-08-30 13:30:38 +1000  425)                                    SEC_PRIV_INVALID, SEC_PRIV_INVALID, 0, des_access,
c352a73badef source3/rpc_server/srv_lsa_nt.c     (Günther Deschner        2009-10-26 23:37:21 +0100  426)                                    &acc_granted, "_lsa_OpenPolicy2" );
8344e945742f source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2008-10-31 10:51:45 -0700  427)       if (!NT_STATUS_IS_OK(status)) {
459dc8f39c08 source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2009-05-18 15:44:03 -0700  428)               return status;
75a5c0b307a7 source3/rpc_server/srv_lsa_nt.c     (Simo Sorce              2003-06-18 15:24:10 +0000  429)       }
75a5c0b307a7 source3/rpc_server/srv_lsa_nt.c     (Simo Sorce              2003-06-18 15:24:10 +0000  430) 
5b412117b2d4 source3/rpc_server/lsa/srv_lsa_nt.c (Günther Deschner        2009-10-30 11:09:52 +0100  431)       status = create_lsa_policy_handle(p->mem_ctx, p,
5b412117b2d4 source3/rpc_server/lsa/srv_lsa_nt.c (Günther Deschner        2009-10-30 11:09:52 +0100  432)                                         LSA_HANDLE_POLICY_TYPE,
5b412117b2d4 source3/rpc_server/lsa/srv_lsa_nt.c (Günther Deschner        2009-10-30 11:09:52 +0100  433)                                         acc_granted,
5b412117b2d4 source3/rpc_server/lsa/srv_lsa_nt.c (Günther Deschner        2009-10-30 11:09:52 +0100  434)                                         get_global_sam_sid(),
5b412117b2d4 source3/rpc_server/lsa/srv_lsa_nt.c (Günther Deschner        2009-10-30 11:09:52 +0100  435)                                         NULL,
5b412117b2d4 source3/rpc_server/lsa/srv_lsa_nt.c (Günther Deschner        2009-10-30 11:09:52 +0100  436)                                         psd,
5b412117b2d4 source3/rpc_server/lsa/srv_lsa_nt.c (Günther Deschner        2009-10-30 11:09:52 +0100  437)                                         r->out.handle);
5b412117b2d4 source3/rpc_server/lsa/srv_lsa_nt.c (Günther Deschner        2009-10-30 11:09:52 +0100  438)       if (!NT_STATUS_IS_OK(status)) {
da3053048c3d source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2001-03-11 00:32:10 +0000  439)               return NT_STATUS_OBJECT_NAME_NOT_FOUND;
5b412117b2d4 source3/rpc_server/lsa/srv_lsa_nt.c (Günther Deschner        2009-10-30 11:09:52 +0100  440)       }
e1951d75fc49 source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2001-02-26 19:31:07 +0000  441) 
b031af348c7d source3/rpc_server/srv_lsa_nt.c     (Andrew Tridgell         2001-08-27 19:46:22 +0000  442)       return NT_STATUS_OK;
e1951d75fc49 source3/rpc_server/srv_lsa_nt.c     (Jeremy Allison          2001-02-26 19:31:07 +0000  443) }


I need to debug it more...
Comment 29 Volker Lendecke 2022-01-18 15:41:48 UTC
Add DBG_DEBUG("transport=%d\n", p->transport); in the check in both master and 4.15. What transport are you coming in via?
Comment 30 Alexander Bokovoy 2022-01-18 16:38:22 UTC
4.16 goes with transport 2:

[2022/01/18 16:28:46, 10, pid=32182, effective(1389000000, 1389000000), real(1389000000, 0), class=rpc_srv] ../../source3/rpc_server/lsa/srv_lsa_nt.c:404(_lsa_OpenPolicy2)
  _lsa_OpenPolicy2: transport=2
[2022/01/18 16:28:46,  4, pid=32182, effective(0, 0), real(0, 0)] ../../source3/smbd/sec_ctx.c:443(pop_sec_ctx)
  pop_sec_ctx (0, 0) - sec_ctx_stack_ndx = 0
[2022/01/18 16:28:46,  5, pid=32182, effective(0, 0), real(0, 0), class=rpc_srv] ../../librpc/rpc/dcesrv_core.c:1941(dcesrv_request)
  dcerpc fault in call lsarpc:2c - DCERPC_FAULT_ACCESS_DENIED
[2022/01/18 16:28:46,  3, pid=32182, effective(0, 0), real(0, 0), class=rpc_srv] ../../source3/rpc_server/rpc_server.c:257(ncacn_terminate_connection)
  ncacn_terminate_connection: Terminating connection - 'dcesrv: NT_STATUS_CONNECTION_DISCONNECTED'

So the ACCESS_DENIED result is expected here because of the check here:

        if (p->transport != NCACN_NP && p->transport != NCALRPC) {
                p->fault_state = DCERPC_FAULT_ACCESS_DENIED;
                return NT_STATUS_ACCESS_DENIED;
        }

To test on 4.15 I need to build a different VM...
Comment 31 Volker Lendecke 2022-01-18 16:44:32 UTC
That's NCACN_IP_TCP. Do you actually come in via TCP?
Comment 32 Alexander Bokovoy 2022-01-18 17:00:29 UTC
Correct -- on the same machine, that is. This is how all our Samba AD domain join and trust code works:

python/samba/join.py:            lsaconn = lsa.lsarpc("ncacn_ip_tcp:%s[%s]" % (ctx.server, binding_options),
python/samba/join.py:        binding_string = "ncacn_ip_tcp:%s[%s]" % (ctx.server, binding_options)
python/samba/join.py:                "ncacn_ip_tcp:%s[%s]" % (ctx.server, binding_options),
python/samba/join.py:        dns_conn = dnsserver.dnsserver("ncacn_ip_tcp:%s[%s]" % (ctx.server, binding_options),
python/samba/join.py:        binding_str = "ncacn_ip_tcp:%s[%s]" % (ctx.server, binding_options)

E.g. this is the same for about a decade and it worked the same way for FreeIPA all the time against s3 LSA server too.

In s4 LSA server we have the same check in 

/* 
  lsa_OpenPolicy2
*/
NTSTATUS dcesrv_lsa_OpenPolicy2(struct dcesrv_call_state *dce_call, TALLOC_CTX *mem_ctx,
                                struct lsa_OpenPolicy2 *r)
{
        enum dcerpc_transport_t transport =
                dcerpc_binding_get_transport(dce_call->conn->endpoint->ep_description);
        NTSTATUS status;
        struct lsa_policy_state *state;
        struct dcesrv_handle *handle;

        if (transport != NCACN_NP && transport != NCALRPC) {
                DCESRV_FAULT(DCERPC_FAULT_ACCESS_DENIED);
        }

...


For example, in python/samba/join.py in cleanup_old_join() we do
...
        if ctx.subdomain:
            binding_options = "sign"
            lsaconn = lsa.lsarpc("ncacn_ip_tcp:%s[%s]" % (ctx.server, binding_options),
                                 ctx.lp, ctx.creds)

            objectAttr = lsa.ObjectAttribute()
            objectAttr.sec_qos = lsa.QosInfo()

            pol_handle = lsaconn.OpenPolicy2('',
                                             objectAttr,
                                             security.SEC_FLAG_MAXIMUM_ALLOWED)

            name = lsa.String()
            name.string = ctx.realm
            info = lsaconn.QueryTrustedDomainInfoByName(pol_handle, name, lsa.LSA_TRUSTED_DOMAIN_INFO_FULL_INFO)

            lsaconn.DeleteTrustedDomain(pol_handle, info.info_ex.sid)

            name = lsa.String()
            name.string = ctx.forest_domain_name
            info = lsaconn.QueryTrustedDomainInfoByName(pol_handle, name, lsa.LSA_TRUSTED_DOMAIN_INFO_FULL_INFO)

            lsaconn.DeleteTrustedDomain(pol_handle, info.info_ex.sid)
Comment 33 Stefan Metzmacher 2022-01-18 17:29:43 UTC
(In reply to Alexander Bokovoy from comment #32)

if ctx.subdomain: is dead code.

join_setup_trusts() has the correct logic and uses ncacn_np.

lsa_OpenPolicy2 never worked over TCP...
Comment 34 Alexander Bokovoy 2022-01-19 06:26:03 UTC
Right, I don't know why I used ncacn_ip_tcp in this test code -- in FreeIPA I have ncacn_np. So this is my own fault.

Ok, so accessing with Kerberos works for NCACN_NP when smbd is used and pipes handled on demand by the smbd. NCACN_NP access does not work when samba-dcerpcd is used instead.

E.g. with smb.conf having

rpc start on demand helpers = true

and smb.service starting smbd all works because port 445 is handled by the smbd then.

And when smb.conf has

rpc start on demand helpers = false

and smb.service is overridden with

systemctl edit smb

<add the following in the editor and save>
[Service]
ExecStart=
ExecStart=/usr/libexec/samba/samba-dcerpcd --libexec-rpcds --foreground

then NCACN_NP access does not work because samba-dcerpcd does not listen on port 445.


[2022/01/19 06:12:55,  3, pid=34061, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:2871(lp_do_section)
  Processing section "[global]"
  doing parameter workgroup = IPA
  doing parameter netbios name = F35
  doing parameter realm = IPA.TEST
  doing parameter kerberos method = dedicated keytab
  doing parameter dedicated keytab file = /etc/samba/samba.keytab
  doing parameter create krb5 conf = no
  doing parameter server role = IPA PRIMARY DOMAIN CONTROLLER
  doing parameter security = user
  doing parameter domain master = yes
  doing parameter max log size = 100000
  doing parameter log file = /var/log/samba/log.%m
  doing parameter passdb backend = ipasam:ldapi://%2fvar%2frun%2fslapd-IPA-TEST.socket
  doing parameter disable spoolss = yes
  doing parameter ldapsam:trusted = yes
  doing parameter ldap ssl = off
  doing parameter ldap suffix = dc=ipa,dc=test
  doing parameter ldap user suffix = cn=users,cn=accounts
  doing parameter ldap group suffix = cn=groups,cn=accounts
  doing parameter ldap machine suffix = cn=computers,cn=accounts
  doing parameter idmap config * : backend = tdb
  doing parameter idmap config * : range = 0 - 0
  doing parameter idmap config IPA : backend = sss
  doing parameter idmap config IPA : range = 1389000000 - 1389200000
  doing parameter max smbd processes = 1000
  doing parameter log level = 10
  doing parameter rpc start on demand helpers = true
[2022/01/19 06:12:55,  4, pid=34061, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:4011(lp_load_ex)
  pm_process() returned Yes
[2022/01/19 06:12:55,  7, pid=34061, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:4346(lp_servicenumber)
  lp_servicenumber: couldn't find homes
[2022/01/19 06:12:55,  8, pid=34061, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:1510(add_a_service)
  add_a_service: Creating snum = 0 for IPC$
[2022/01/19 06:12:55, 10, pid=34061, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:1552(hash_a_service)
  hash_a_service: creating servicehash
[2022/01/19 06:12:55, 10, pid=34061, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:1560(hash_a_service)
  hash_a_service: hashing index 0 for service name IPC$
[2022/01/19 06:12:55,  3, pid=34061, effective(0, 0), real(0, 0)] ../../source3/param/loadparm.c:1672(lp_add_ipc)
  adding IPC service
[2022/01/19 06:12:55,  5, pid=34061, effective(0, 0), real(0, 0)] ../../source3/auth/auth_generic.c:250(auth3_generate_session_info_pac)
  ../../source3/auth/auth_generic.c:250OK: user: admin domain: IPA client: 192.168.122.19
[2022/01/19 06:12:55,  4, pid=34061, effective(0, 0), real(0, 0), class=auth_audit] ../../auth/auth_log.c:740(log_successful_authz_event_human_readable)
  Successful AuthZ: [DCE/RPC,krb5] user [IPA]\[admin] [S-1-5-21-3703471042-164549623-3970024037-500] at [Wed, 19 Jan 2022 06:12:55.052919 UTC] Remote host [ipv4:192.168.122.19:49074] local host [ipv4:192.168.122.19:445]
  {"timestamp": "2022-01-19T06:12:55.052946+0000", "type": "Authorization", "Authorization": {"version": {"major": 1, "minor": 1}, "localAddress": "ipv4:192.168.122.19:445", "remoteAddress": "ipv4:192.168.122.19:49074", "serviceDescription": "DCE/RPC", "
authType": "krb5", "domain": "IPA", "account": "admin", "sid": "S-1-5-21-3703471042-164549623-3970024037-500", "sessionId": "92e07ae1-e108-46ed-9dca-167cda85326e", "logonServer": "F35", "transportProtection": "SEAL", "accountFlags": "0x00000010"}}
[2022/01/19 06:12:55, 10, pid=34061, effective(0, 0), real(0, 0)] ../../librpc/rpc/dcerpc_util.c:399(dcerpc_pull_auth_trailer)
  dcerpc_pull_auth_trailer: auth_pad_length 12
[2022/01/19 06:12:55, 10, pid=34061, effective(0, 0), real(0, 0), class=auth] ../../auth/kerberos/gssapi_helper.c:305(gssapi_unseal_packet)
  Unsealed 128 bytes, with 76 bytes header/signature.
[2022/01/19 06:12:55,  4, pid=34061, effective(0, 0), real(0, 0)] ../../source3/smbd/sec_ctx.c:206(push_sec_ctx)
  push_sec_ctx(0, 0) : sec_ctx_stack_ndx = 1
[2022/01/19 06:12:55,  4, pid=34061, effective(0, 0), real(0, 0)] ../../source3/smbd/sec_ctx.c:317(set_sec_ctx_internal)
  setting sec ctx (1389000000, 1389000000) - sec_ctx_stack_ndx = 1
[2022/01/19 06:12:55,  5, pid=34061, effective(0, 0), real(0, 0)] ../../libcli/security/security_token.c:51(security_token_debug)
  Security token SIDs (9):
    SID[  0]: S-1-5-21-3703471042-164549623-3970024037-500
    SID[  1]: S-1-5-21-3703471042-164549623-3970024037-512
    SID[  2]: S-1-18-1
    SID[  3]: S-1-1-0
    SID[  4]: S-1-5-2
    SID[  5]: S-1-5-11
    SID[  6]: S-1-5-32-544
    SID[  7]: S-1-22-1-1389000000
    SID[  8]: S-1-22-2-1389000000
   Privileges (0x        1FFFFFF0):
    Privilege[  0]: SeMachineAccountPrivilege
    Privilege[  1]: SeTakeOwnershipPrivilege
    Privilege[  2]: SeBackupPrivilege
    Privilege[  3]: SeRestorePrivilege
    Privilege[  4]: SeRemoteShutdownPrivilege
    Privilege[  5]: SePrintOperatorPrivilege
    Privilege[  6]: SeAddUsersPrivilege
    Privilege[  7]: SeDiskOperatorPrivilege
    Privilege[  8]: SeSecurityPrivilege
    Privilege[  9]: SeSystemtimePrivilege
    Privilege[ 10]: SeShutdownPrivilege
    Privilege[ 11]: SeDebugPrivilege
    Privilege[ 12]: SeSystemEnvironmentPrivilege
    Privilege[ 13]: SeSystemProfilePrivilege
    Privilege[ 14]: SeProfileSingleProcessPrivilege
    Privilege[ 15]: SeIncreaseBasePriorityPrivilege
    Privilege[ 16]: SeLoadDriverPrivilege
    Privilege[ 17]: SeCreatePagefilePrivilege
    Privilege[ 18]: SeIncreaseQuotaPrivilege
    Privilege[ 19]: SeChangeNotifyPrivilege
    Privilege[ 20]: SeUndockPrivilege
    Privilege[ 21]: SeManageVolumePrivilege
    Privilege[ 22]: SeImpersonatePrivilege
    Privilege[ 23]: SeCreateGlobalPrivilege
    Privilege[ 24]: SeEnableDelegationPrivilege
   Rights (0x               0):
[2022/01/19 06:12:55,  5, pid=34061, effective(0, 0), real(0, 0)] ../../source3/auth/token_util.c:873(debug_unix_user_token)
  UNIX token of user 1389000000
  Primary group is 1389000000 and contains 1 supplementary groups
  Group[  0]: 1389000000
[2022/01/19 06:12:55,  5, pid=34061, effective(1389000000, 1389000000), real(1389000000, 0)] ../../source3/smbd/uid.c:522(smbd_become_authenticated_pipe_user)
  Impersonated user: uid=(1389000000,1389000000), gid=(0,1389000000)
[2022/01/19 06:12:55,  1, pid=34061, effective(1389000000, 1389000000), real(1389000000, 0), class=rpc_parse] ../../librpc/ndr/ndr.c:484(ndr_print_function_debug)
       lsa_OpenPolicy2: struct lsa_OpenPolicy2
          in: struct lsa_OpenPolicy2
              system_name              : *
                  system_name              : ''
              attr                     : *
                  attr: struct lsa_ObjectAttribute
                      len                      : 0x00000000 (0)
                      root_dir                 : NULL
                      object_name              : NULL
                      attributes               : 0x00000000 (0)
                      sec_desc                 : NULL
                      sec_qos                  : *
                          sec_qos: struct lsa_QosInfo
                              len                      : 0x00000000 (0)
                              impersonation_level      : 0x0000 (0)
                              context_mode             : 0x00 (0)
                              effective_only           : 0x00 (0)
              access_mask              : 0x00000029 (41)
                     1: LSA_POLICY_VIEW_LOCAL_INFORMATION
                     0: LSA_POLICY_VIEW_AUDIT_INFORMATION
                     0: LSA_POLICY_GET_PRIVATE_INFORMATION
                     1: LSA_POLICY_TRUST_ADMIN   
                     0: LSA_POLICY_CREATE_ACCOUNT
                     1: LSA_POLICY_CREATE_SECRET 
                     0: LSA_POLICY_CREATE_PRIVILEGE
                     0: LSA_POLICY_SET_DEFAULT_QUOTA_LIMITS
                     0: LSA_POLICY_SET_AUDIT_REQUIREMENTS
                     0: LSA_POLICY_AUDIT_LOG_ADMIN
                     0: LSA_POLICY_SERVER_ADMIN  
                     0: LSA_POLICY_LOOKUP_NAMES  
                     0: LSA_POLICY_NOTIFICATION  
[2022/01/19 06:12:55, 10, pid=34061, effective(1389000000, 1389000000), real(1389000000, 0), class=rpc_srv] ../../source3/rpc_server/lsa/srv_lsa_nt.c:404(_lsa_OpenPolicy2)
  _lsa_OpenPolicy2: transport=1
[2022/01/19 06:12:55,  4, pid=34061, effective(1389000000, 1389000000), real(1389000000, 0), class=rpc_srv] ../../source3/rpc_server/srv_access_check.c:118(access_check_object)
  _lsa_OpenPolicy2: access GRANTED (requested: 0x00000029, granted: 0x00000029)
[2022/01/19 06:12:55,  4, pid=34061, effective(0, 0), real(0, 0)] ../../source3/smbd/sec_ctx.c:443(pop_sec_ctx)
  pop_sec_ctx (0, 0) - sec_ctx_stack_ndx = 0
[2022/01/19 06:12:55,  1, pid=34061, effective(0, 0), real(0, 0), class=rpc_parse] ../../librpc/ndr/ndr.c:484(ndr_print_function_debug)
       lsa_OpenPolicy2: struct lsa_OpenPolicy2
          out: struct lsa_OpenPolicy2
              handle                   : *
                  handle: struct policy_handle
                      handle_type              : 0x00000001 (1)
                      uuid                     : 7973aefd-ea51-4ee7-9830-87d68f422458
              result                   : NT_STATUS_OK
[2022/01/19 06:12:55, 10, pid=34061, effective(0, 0), real(0, 0), class=auth] ../../auth/kerberos/gssapi_helper.c:208(gssapi_seal_packet)
  Sealed 32 bytes, and got 76 bytes header/signature.
[2022/01/19 06:12:55,  3, pid=34061, effective(0, 0), real(0, 0), class=rpc_srv] ../../source3/rpc_server/rpc_server.c:257(ncacn_terminate_connection)
  ncacn_terminate_connection: Terminating connection - 'dcesrv: NT_STATUS_CONNECTION_DISCONNECTED'
[2022/01/19 06:12:55,  1, pid=34061, effective(0, 0), real(0, 0), class=rpc_parse] ../../librpc/ndr/ndr.c:435(ndr_print_debug)
       &worker->status: struct rpc_worker_status
          server_index             : 0x01 (1)
          worker_index             : 0x00 (0)
          num_clients              : 0x00000000 (0)
[2022/01/19 06:12:55, 10, pid=34061, effective(0, 0), real(0, 0)] ../../lib/messaging/messages_dgm.c:1463(messaging_dgm_send)
  messaging_dgm_send: Sending message to 34051
[2022/01/19 06:13:55, 10, pid=34061, effective(0, 0), real(0, 0)] ../../source3/lib/messages.c:421(messaging_recv_cb)
  messaging_recv_cb: Received message 0xd len 0 (num_fds:0) from 34051
[2022/01/19 06:13:55,  5, pid=34061, effective(0, 0), real(0, 0), class=rpc_srv] ../../librpc/rpc/dcesrv_core.c:2559(dcesrv_shutdown_ep_server)
  dcesrv_shutdown_ep_server: Shutting down DCE/RPC endpoint server 'lsarpc'
[2022/01/19 06:13:55,  5, pid=34061, effective(0, 0), real(0, 0), class=rpc_srv] ../../librpc/rpc/dcesrv_core.c:2559(dcesrv_shutdown_ep_server)
  dcesrv_shutdown_ep_server: Shutting down DCE/RPC endpoint server 'samr'
[2022/01/19 06:13:55,  5, pid=34061, effective(0, 0), real(0, 0), class=rpc_srv] ../../librpc/rpc/dcesrv_core.c:2559(dcesrv_shutdown_ep_server)
  dcesrv_shutdown_ep_server: Shutting down DCE/RPC endpoint server 'dssetup'
[2022/01/19 06:13:55,  5, pid=34061, effective(0, 0), real(0, 0), class=rpc_srv] ../../librpc/rpc/dcesrv_core.c:2559(dcesrv_shutdown_ep_server)
  dcesrv_shutdown_ep_server: Shutting down DCE/RPC endpoint server 'netlogon'


The only change I need to do on FreeIPA side for Samba 4.16 is to update smb.conf to remove

rpc_server:epmapper = external
rpc_server:lsarpc = external
rpc_server:lsass = external
rpc_server:lsasd = external
rpc_server:samr = external
rpc_server:netlogon = external
rpc_server:tcpip = yes
rpc_daemon:epmd = fork
rpc_daemon:lsasd = fork


The client code still requests upper-cased hostname-based principal which I consider wrong but IPA KDC accepts it.

Starting GENSEC mechanism spnego
Starting GENSEC submechanism gssapi_krb5
GSSAPI credentials for admin@IPA.TEST will expire in 86400 secs
[34140] 1642573287.287250: Getting credentials admin@IPA.TEST -> host/F35.IPA.TEST@IPA.TEST using ccache MEMORY:0x55fafa167b40
[34140] 1642573287.287251: Retrieving admin@IPA.TEST -> krb5_ccache_conf_data/start_realm@X-CACHECONF: from MEMORY:0x55fafa167b40 with result: -1765328243/Matching credential not found
[34140] 1642573287.287252: Retrieving admin@IPA.TEST -> host/F35.IPA.TEST@IPA.TEST from MEMORY:0x55fafa167b40 with result: -1765328243/Matching credential not found
[34140] 1642573287.287253: Retrieving admin@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST from MEMORY:0x55fafa167b40 with result: 0/Success
[34140] 1642573287.287254: Starting with TGT for client realm: admin@IPA.TEST -> krbtgt/IPA.TEST@IPA.TEST
[34140] 1642573287.287255: Requesting tickets for host/F35.IPA.TEST@IPA.TEST, referrals on
[34140] 1642573287.287256: Generated subkey for TGS request: aes256-cts/CEC4
[34140] 1642573287.287257: etypes requested in TGS request: aes256-cts, aes256-sha2, camellia256-cts, aes128-sha2, aes128-cts, camellia128-cts
[34140] 1642573287.287259: Encoding request body and padata into FAST request
[34140] 1642573287.287260: Sending request (1837 bytes) to IPA.TEST
[34140] 1642573287.287261: Initiating TCP connection to stream 192.168.122.19:88
[34140] 1642573287.287262: Sending TCP request to stream 192.168.122.19:88
[34140] 1642573287.287263: Received answer (1760 bytes) from stream 192.168.122.19:88
[34140] 1642573287.287264: Terminating TCP connection to stream 192.168.122.19:88
[34140] 1642573287.287265: Response was from primary KDC
[34140] 1642573287.287266: Decoding FAST response
[34140] 1642573287.287267: FAST reply key: aes256-cts/040F
[34140] 1642573287.287268: TGS reply is for admin@IPA.TEST -> host/F35.IPA.TEST@IPA.TEST with session key aes256-cts/62D6
[34140] 1642573287.287269: TGS request result: 0/Success
[34140] 1642573287.287270: Received creds for desired service host/F35.IPA.TEST@IPA.TEST
[34140] 1642573287.287271: Storing admin@IPA.TEST -> host/F35.IPA.TEST@IPA.TEST in MEMORY:0x55fafa167b40
[34140] 1642573287.287273: Creating authenticator for admin@IPA.TEST -> host/F35.IPA.TEST@IPA.TEST, seqnum 698560765, subkey aes256-cts/18C7, session key aes256-cts/62D6

Here is a set of requests done during the test run which show we have no problem with upper-cased hostname:

Jan 19 06:12:54 f35.ipa.test krb5kdc[28222](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia128-cts-cmac(25)}) 192.168.122.19: ISSUE: authtime 1642572774, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, admin@IPA.TEST for krbtgt/IPA.TEST@IPA.TEST
Jan 19 06:12:54 f35.ipa.test krb5kdc[28222](info): closing down fd 4
Jan 19 06:12:54 f35.ipa.test krb5kdc[28222](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia128-cts-cmac(25)}) 192.168.122.19: ISSUE: authtime 1642572774, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, admin@IPA.TEST for cifs/f35.ipa.test@IPA.TEST
Jan 19 06:12:54 f35.ipa.test krb5kdc[28222](info): closing down fd 4
Jan 19 06:12:54 f35.ipa.test krb5kdc[28221](info): TGS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia128-cts-cmac(25)}) 192.168.122.19: ISSUE: authtime 1642572774, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, admin@IPA.TEST for host/f35.ipa.test@IPA.TEST
Jan 19 06:12:54 f35.ipa.test krb5kdc[28221](info): closing down fd 4
Jan 19 06:13:06 f35.ipa.test krb5kdc[28221](info): AS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17), camellia128-cts-cmac(25)}) 192.168.122.19: NEEDED_PREAUTH: host/f35.ipa.test@IPA.TEST for krbtgt/IPA.TEST@IPA.TEST, Additional pre-authentication required
Jan 19 06:13:06 f35.ipa.test krb5kdc[28221](info): closing down fd 4

However, I think we should fix Samba code in the manner proposed by Jeremy. I will work on the test based on my code during this weekend.