Bug 14979 - problem when winbind renews Kerberos
Summary: problem when winbind renews Kerberos
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Winbind (show other bugs)
Version: 4.15.5
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Jule Anger
QA Contact: Samba QA Contact
URL: https://gitlab.com/samba-team/samba/-...
Keywords:
Depends on:
Blocks:
 
Reported: 2022-02-14 18:31 UTC by Jason Keltz
Modified: 2022-09-28 15:40 UTC (History)
3 users (show)

See Also:


Attachments
patch for 4.16 (13.69 KB, patch)
2022-02-24 08:48 UTC, Andreas Schneider
metze: review+
scabrero: review+
Details
patch for 4.15 (13.69 KB, patch)
2022-02-24 08:50 UTC, Andreas Schneider
metze: review+
scabrero: review+
Details
patch for 4.14 (13.69 KB, patch)
2022-02-24 10:26 UTC, Samuel Cabrero
asn: review+
metze: review+
Details
patch for 4.16 (part2) (11.65 KB, patch)
2022-07-12 15:39 UTC, Samuel Cabrero
asn: review+
Details
patch for 4.15 (part2) (11.65 KB, patch)
2022-07-12 15:40 UTC, Samuel Cabrero
asn: review+
Details
patch for 4.14 (part2) (11.65 KB, patch)
2022-07-12 15:40 UTC, Samuel Cabrero
asn: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jason Keltz 2022-02-14 18:31:01 UTC
We have 2 samba AD DC machines running 4.15.5, a samba file server running 4.15.5, 3 NFS servers running 4.15.5, and a whole bunch of clients running 4.15.5.  

When I login to my CentOS 7 workstation, and I do a klist, I see that the default Kerberos principal is "jas@AD.EECS.YORKU.CA" which is correct.  Everything works perfectly.  At some point during the day, when I open a new Terminal window, I end up getting a "permission denied" error.  At this point, if I do a klist, I see that my default principal has CHANGED to jas\@EECSYORKUCA@AD.EECS.YORKU.CA.  My connection to the NFS server fails because the principal changed.  I *believe* it is winbind making this change.  My client smb.conf configuration has been working for quite some time...

[global]
workgroup = EECSYORKUCA
security = ADS
realm = AD.EECS.YORKU.CA
dedicated keytab file = /etc/krb5.keytab
kerberos method = secrets and keytab
idmap config * : backend = tdb
idmap config * : range = 1000000-1999999
idmap config EECSYORKUCA : backend = ad
idmap config EECSYORKUCA : schema_mode = rfc2307
idmap config EECSYORKUCA : range = 1000-999999
idmap config EECSYORKUCA : unix_primary_group = yes
idmap config EECSYORKUCA : unix_nss_info = yes
winbind refresh tickets = yes
winbind offline logon = yes
winbind nss info = rfc2307
winbind use default domain = yes
winbind enum users  = no
winbind enum groups = no

We didn't use to have this problem at all and the only thing that has changed is the Samba version.  I'd really like to solve this problem though, so if you tell me what debugging to enable or what logging to turn on, I would be more than happy to help.  I would greatly appreciate if we could fix this.
Comment 1 Stefan Metzmacher 2022-02-15 09:46:58 UTC
What kerberos library is in use?

Does "winbind use krb5 enterprise principals = no" change things?
Comment 2 Jason Keltz 2022-02-15 13:07:17 UTC
While CentOS 7 has MIT Kerberos, we're using the built in Samba kerberos.  

% ldd winbindd | grep -i krb


	libkrb5samba-samba4.so => /xsys/pkg/samba-4.15.5/lib/private/libkrb5samba-samba4.so (0x00007f76e7459000)
	libkrb5-samba4.so.26 => /xsys/pkg/samba-4.15.5/lib/private/libkrb5-samba4.so.26 (0x00007f76e24c1000)
	libauthkrb5-samba4.so => /xsys/pkg/samba-4.15.5/lib/private/libauthkrb5-samba4.so (0x00007f76e1a6a000)
	libndr-krb5pac.so.0 => /xsys/pkg/samba-4.15.5/lib/libndr-krb5pac.so.0 (0x00007f76df6bd000)
	libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f76d479a000)
	libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f76d44b1000)
	libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f76d3e6a000)

I will configure  "winbind use krb5 enterprise principals = no" on my workstation, and will report back the status. Thank you!
Comment 3 Jason Keltz 2022-02-15 21:19:39 UTC
For my full work day, I haven't seen the problem occur one time. This "winbind use krb5 enterprise principals = no" may be the solution.  I'll see what happens when I return to my workstation tomorrow morning.  I've seen the description in the man page, but it's not 100% clear what this is doing.  It's also not clear why this would be required now when it wasn't before, but I suspect there may have been a change a few versions back....  so far, very positive. thank you Stefan.
Comment 4 Jason Keltz 2022-02-16 19:25:27 UTC
Everything continued to function today flawlessly.
As I understand it, the option for "winbind use krb5 enterprise principals" changed from default "no" to "yes" in 4.15.  Since we didn't hop on 4.15 right at beginning, I now understand that my problems started occurring when we upgraded to Samba 4.15.
In the 4.15 docs, it says however that the option for "winbind use krb5 enterprise principals" will be deprecated in a future release.  As a result, it is important for me to understand why my system doesn't work with the default configuration, having this feature enabled.  Can you please help me understand whether this is a bug on my system (any thoughts on how to fix it? is it something I need to add to krb5.conf?), or a bug in Samba?  It is interesting the initial principal gets changed.
Thank you.
Comment 5 Stefan Metzmacher 2022-02-23 08:25:18 UTC
https://gitlab.com/samba-team/samba/-/merge_requests/2405 will fix the problem
Comment 6 Jason Keltz 2022-02-23 10:34:22 UTC
I'm very thankful that the bug was identified and patched. However, does this patch address why, with enterprise credentials enabled, the initial principal is not of the enterprise form? Is that fixed as well?
And can someone explain why the enterprise principal versus the standard form is the new norm? jas@AD.EECS.YORKU.CA just seems more logical as an identifier for me versus jas\@EECSYORKUCA@AD.EECS.YORKU.CA?
Comment 7 Stefan Metzmacher 2022-02-23 11:53:59 UTC
(In reply to Jason Keltz from comment #6)

The enterprise principal is only used in the initial kinit (AS-REQ),
that form is much more reliable to trusted domains and simplifies the
routing.

I guess the authentication is done as EECSYORKUCA\jas.

So we send jas\@EECSYORKUCA@AD.EECS.YORKU.CA without knowing that
EECSYORKUCA and AD.EECS.YORKU.CA belong together, as that's the
job of the domain controllers.

We just convert EECSYORKUCA\jas into jas@EECSYORKUCA
and append our local realm to form the enterprise principal.

We also set the canonicalize bit in the AS-REQ, so we'll
get back a TGT for jas@AD.EECS.YORKU.CA.

AD.EECS.YORKU.CA would have a trust to SAMBA.ORG

and I'd login as SAMBA\metze we would start with

metze\@SAMBA@AD.EECS.YORKU.CA ending the AS-REQ to a KDC of
AD.EECS.YORKU.CA, but that will detect that SAMBA belongs to a trust
with SAMBA.ORG and response with ERR_WRONG_REALM redirecting
us to SAMBA.ORG.

Then we send an AS-REQ for metze\@SAMBA@SAMBA.ORG to a KDC of SAMBA.ORG,
which would then response with a ticket for metze@SAMBA.ORG

If there would be more transitive hops in between, we hop would be
able to redirect us one step closer to the users real realm.
Comment 8 Jason Keltz 2022-02-23 14:20:06 UTC
(In reply to Stefan Metzmacher from comment #7)

Thank you very much for the description.  On my side, the authentication is done as just "jas" but because "winbind use default domain = yes", it is authenticating as EECSYORKUCA\jas.  Your description helped me to understand the process much better.

I look forward to testing in a Samba release soon.
Comment 9 Samba QA Contact 2022-02-23 16:18:07 UTC
This bug was referenced in samba master:

00b1f44a7e8f66976757535bcbc6bea97fb1c29f
0f4f330773d272b4d28ff3ba5a41bdd4ba569c8b
8246ccc23d064147412bb3475e6431a9fffc0d27
Comment 10 Andreas Schneider 2022-02-24 08:48:18 UTC
Created attachment 17177 [details]
patch for 4.16
Comment 11 Andreas Schneider 2022-02-24 08:50:04 UTC
Created attachment 17178 [details]
patch for 4.15
Comment 12 Stefan Metzmacher 2022-02-24 09:47:57 UTC
Don't we want this in 4.14 too?
Comment 13 Samuel Cabrero 2022-02-24 09:58:20 UTC
(In reply to Stefan Metzmacher from comment #12)
Probably 4.13 too, enterprise principals were introduced in 4.12. I will do the backports.
Comment 14 Stefan Metzmacher 2022-02-24 10:08:09 UTC
(In reply to Samuel Cabrero from comment #13)

Thanks, but only 4.14 will get it applied, 4.13 is in security mode already
Comment 15 Samuel Cabrero 2022-02-24 10:26:46 UTC
Created attachment 17179 [details]
patch for 4.14
Comment 16 Jule Anger 2022-02-24 13:55:59 UTC
Pushed to autobuild-v4-{16,15,14}-test.
Comment 17 Samba QA Contact 2022-02-25 11:37:05 UTC
This bug was referenced in samba v4-14-test:

f5672ef042bdfbf0aa584f5a3f6f3df6e54b2017
989153501516ad4c059103db8955259ae988b763
34ade9eab0a07cf2be3059b3b9c5ae9af7f878f2
Comment 18 Samba QA Contact 2022-02-25 18:09:03 UTC
This bug was referenced in samba v4-16-test:

85fdd88e3cabbd405ddfda32b0b2b4e7eeee673e
e3efe2d0de20216ea7ecbc8e41ae8c6968507d29
4346dac73a4e74b1e286346f1e6d867a8369f5fd
Comment 19 Samba QA Contact 2022-02-27 10:32:03 UTC
This bug was referenced in samba v4-15-test:

29218c61b0ad240fa7c88f93a914063fb28f52e0
d0d70d9f6309c2779d6c912010ebd1a9ca571b87
4d91f6dafd164cee7118410232aef8e7a762da71
Comment 20 Jule Anger 2022-02-27 16:20:15 UTC
Closing out bug report.

Thanks!
Comment 21 Samba QA Contact 2022-03-01 08:55:13 UTC
This bug was referenced in samba v4-16-stable (Release samba-4.16.0rc4):

85fdd88e3cabbd405ddfda32b0b2b4e7eeee673e
e3efe2d0de20216ea7ecbc8e41ae8c6968507d29
4346dac73a4e74b1e286346f1e6d867a8369f5fd
Comment 22 Samba QA Contact 2022-03-15 13:26:23 UTC
This bug was referenced in samba v4-15-stable (Release samba-4.15.6):

29218c61b0ad240fa7c88f93a914063fb28f52e0
d0d70d9f6309c2779d6c912010ebd1a9ca571b87
4d91f6dafd164cee7118410232aef8e7a762da71
Comment 23 Samba QA Contact 2022-04-04 12:51:23 UTC
This bug was referenced in samba v4-14-stable (Release samba-4.14.13):

f5672ef042bdfbf0aa584f5a3f6f3df6e54b2017
989153501516ad4c059103db8955259ae988b763
34ade9eab0a07cf2be3059b3b9c5ae9af7f878f2
Comment 24 Samba QA Contact 2022-07-12 12:39:04 UTC
This bug was referenced in samba master:

8bef8e3de9fc96ff45319f80529e878977563f3a
116af0df4f74aa450cbb77c79f8cac4bfc288631
Comment 25 Samuel Cabrero 2022-07-12 15:28:58 UTC
The issue was not completely fixed by the released patches and winbind still fails to refresh the tickets in certain scenarios or AD topologies. Details in https://gitlab.com/samba-team/samba/-/merge_requests/2610.
Comment 26 Samuel Cabrero 2022-07-12 15:39:39 UTC
Created attachment 17414 [details]
patch for 4.16 (part2)
Comment 27 Samuel Cabrero 2022-07-12 15:40:28 UTC
Created attachment 17415 [details]
patch for 4.15 (part2)
Comment 28 Samuel Cabrero 2022-07-12 15:40:59 UTC
Created attachment 17416 [details]
patch for 4.14 (part2)
Comment 29 Jule Anger 2022-07-18 07:56:11 UTC
Pushed to autobuild-v4-{16,15}-test.
4.14 is in security mode already.
Comment 30 Samba QA Contact 2022-07-18 09:41:04 UTC
This bug was referenced in samba v4-16-test:

e388fe2b70120253be6eed5ed7e53cec9ea0b0ef
89b914b3c515f4fc91f8870031c359652ebb77a7
Comment 31 Samba QA Contact 2022-07-18 10:37:03 UTC
This bug was referenced in samba v4-15-test:

e7ae7cba1361fec80df015ccc8263b2133cd877a
ca5abc39c1d1f8d3bfa7bee79a1cf0b1944fc85d
Comment 32 Samba QA Contact 2022-07-18 11:17:52 UTC
This bug was referenced in samba v4-16-stable (Release samba-4.16.3):

e388fe2b70120253be6eed5ed7e53cec9ea0b0ef
89b914b3c515f4fc91f8870031c359652ebb77a7
Comment 33 Jule Anger 2022-07-31 18:04:11 UTC
Closing out bug report.

Thanks!
Comment 34 Samba QA Contact 2022-09-28 15:40:35 UTC
This bug was referenced in samba v4-15-stable (Release samba-4.15.10):

e7ae7cba1361fec80df015ccc8263b2133cd877a
ca5abc39c1d1f8d3bfa7bee79a1cf0b1944fc85d