Bug 14979 - problem when winbind renews Kerberos
Summary: problem when winbind renews Kerberos
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Winbind (show other bugs)
Version: 4.15.5
Hardware: All All
: P5 normal (vote)
Target Milestone: ---
Assignee: Jule Anger
QA Contact: Samba QA Contact
URL: https://gitlab.com/samba-team/samba/-...
Depends on:
Reported: 2022-02-14 18:31 UTC by Jason Keltz
Modified: 2022-09-28 15:40 UTC (History)
3 users (show)

See Also:

patch for 4.16 (13.69 KB, patch)
2022-02-24 08:48 UTC, Andreas Schneider
metze: review+
scabrero: review+
patch for 4.15 (13.69 KB, patch)
2022-02-24 08:50 UTC, Andreas Schneider
metze: review+
scabrero: review+
patch for 4.14 (13.69 KB, patch)
2022-02-24 10:26 UTC, Samuel Cabrero
asn: review+
metze: review+
patch for 4.16 (part2) (11.65 KB, patch)
2022-07-12 15:39 UTC, Samuel Cabrero
asn: review+
patch for 4.15 (part2) (11.65 KB, patch)
2022-07-12 15:40 UTC, Samuel Cabrero
asn: review+
patch for 4.14 (part2) (11.65 KB, patch)
2022-07-12 15:40 UTC, Samuel Cabrero
asn: review+

Note You need to log in before you can comment on or make changes to this bug.
Description Jason Keltz 2022-02-14 18:31:01 UTC
We have 2 samba AD DC machines running 4.15.5, a samba file server running 4.15.5, 3 NFS servers running 4.15.5, and a whole bunch of clients running 4.15.5.  

When I login to my CentOS 7 workstation, and I do a klist, I see that the default Kerberos principal is "jas@AD.EECS.YORKU.CA" which is correct.  Everything works perfectly.  At some point during the day, when I open a new Terminal window, I end up getting a "permission denied" error.  At this point, if I do a klist, I see that my default principal has CHANGED to jas\@EECSYORKUCA@AD.EECS.YORKU.CA.  My connection to the NFS server fails because the principal changed.  I *believe* it is winbind making this change.  My client smb.conf configuration has been working for quite some time...

workgroup = EECSYORKUCA
security = ADS
dedicated keytab file = /etc/krb5.keytab
kerberos method = secrets and keytab
idmap config * : backend = tdb
idmap config * : range = 1000000-1999999
idmap config EECSYORKUCA : backend = ad
idmap config EECSYORKUCA : schema_mode = rfc2307
idmap config EECSYORKUCA : range = 1000-999999
idmap config EECSYORKUCA : unix_primary_group = yes
idmap config EECSYORKUCA : unix_nss_info = yes
winbind refresh tickets = yes
winbind offline logon = yes
winbind nss info = rfc2307
winbind use default domain = yes
winbind enum users  = no
winbind enum groups = no

We didn't use to have this problem at all and the only thing that has changed is the Samba version.  I'd really like to solve this problem though, so if you tell me what debugging to enable or what logging to turn on, I would be more than happy to help.  I would greatly appreciate if we could fix this.
Comment 1 Stefan Metzmacher 2022-02-15 09:46:58 UTC
What kerberos library is in use?

Does "winbind use krb5 enterprise principals = no" change things?
Comment 2 Jason Keltz 2022-02-15 13:07:17 UTC
While CentOS 7 has MIT Kerberos, we're using the built in Samba kerberos.  

% ldd winbindd | grep -i krb

	libkrb5samba-samba4.so => /xsys/pkg/samba-4.15.5/lib/private/libkrb5samba-samba4.so (0x00007f76e7459000)
	libkrb5-samba4.so.26 => /xsys/pkg/samba-4.15.5/lib/private/libkrb5-samba4.so.26 (0x00007f76e24c1000)
	libauthkrb5-samba4.so => /xsys/pkg/samba-4.15.5/lib/private/libauthkrb5-samba4.so (0x00007f76e1a6a000)
	libndr-krb5pac.so.0 => /xsys/pkg/samba-4.15.5/lib/libndr-krb5pac.so.0 (0x00007f76df6bd000)
	libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f76d479a000)
	libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f76d44b1000)
	libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f76d3e6a000)

I will configure  "winbind use krb5 enterprise principals = no" on my workstation, and will report back the status. Thank you!
Comment 3 Jason Keltz 2022-02-15 21:19:39 UTC
For my full work day, I haven't seen the problem occur one time. This "winbind use krb5 enterprise principals = no" may be the solution.  I'll see what happens when I return to my workstation tomorrow morning.  I've seen the description in the man page, but it's not 100% clear what this is doing.  It's also not clear why this would be required now when it wasn't before, but I suspect there may have been a change a few versions back....  so far, very positive. thank you Stefan.
Comment 4 Jason Keltz 2022-02-16 19:25:27 UTC
Everything continued to function today flawlessly.
As I understand it, the option for "winbind use krb5 enterprise principals" changed from default "no" to "yes" in 4.15.  Since we didn't hop on 4.15 right at beginning, I now understand that my problems started occurring when we upgraded to Samba 4.15.
In the 4.15 docs, it says however that the option for "winbind use krb5 enterprise principals" will be deprecated in a future release.  As a result, it is important for me to understand why my system doesn't work with the default configuration, having this feature enabled.  Can you please help me understand whether this is a bug on my system (any thoughts on how to fix it? is it something I need to add to krb5.conf?), or a bug in Samba?  It is interesting the initial principal gets changed.
Thank you.
Comment 5 Stefan Metzmacher 2022-02-23 08:25:18 UTC
https://gitlab.com/samba-team/samba/-/merge_requests/2405 will fix the problem
Comment 6 Jason Keltz 2022-02-23 10:34:22 UTC
I'm very thankful that the bug was identified and patched. However, does this patch address why, with enterprise credentials enabled, the initial principal is not of the enterprise form? Is that fixed as well?
And can someone explain why the enterprise principal versus the standard form is the new norm? jas@AD.EECS.YORKU.CA just seems more logical as an identifier for me versus jas\@EECSYORKUCA@AD.EECS.YORKU.CA?
Comment 7 Stefan Metzmacher 2022-02-23 11:53:59 UTC
(In reply to Jason Keltz from comment #6)

The enterprise principal is only used in the initial kinit (AS-REQ),
that form is much more reliable to trusted domains and simplifies the

I guess the authentication is done as EECSYORKUCA\jas.

So we send jas\@EECSYORKUCA@AD.EECS.YORKU.CA without knowing that
EECSYORKUCA and AD.EECS.YORKU.CA belong together, as that's the
job of the domain controllers.

We just convert EECSYORKUCA\jas into jas@EECSYORKUCA
and append our local realm to form the enterprise principal.

We also set the canonicalize bit in the AS-REQ, so we'll
get back a TGT for jas@AD.EECS.YORKU.CA.

AD.EECS.YORKU.CA would have a trust to SAMBA.ORG

and I'd login as SAMBA\metze we would start with

metze\@SAMBA@AD.EECS.YORKU.CA ending the AS-REQ to a KDC of
AD.EECS.YORKU.CA, but that will detect that SAMBA belongs to a trust
with SAMBA.ORG and response with ERR_WRONG_REALM redirecting
us to SAMBA.ORG.

Then we send an AS-REQ for metze\@SAMBA@SAMBA.ORG to a KDC of SAMBA.ORG,
which would then response with a ticket for metze@SAMBA.ORG

If there would be more transitive hops in between, we hop would be
able to redirect us one step closer to the users real realm.
Comment 8 Jason Keltz 2022-02-23 14:20:06 UTC
(In reply to Stefan Metzmacher from comment #7)

Thank you very much for the description.  On my side, the authentication is done as just "jas" but because "winbind use default domain = yes", it is authenticating as EECSYORKUCA\jas.  Your description helped me to understand the process much better.

I look forward to testing in a Samba release soon.
Comment 9 Samba QA Contact 2022-02-23 16:18:07 UTC
This bug was referenced in samba master:

Comment 10 Andreas Schneider 2022-02-24 08:48:18 UTC
Created attachment 17177 [details]
patch for 4.16
Comment 11 Andreas Schneider 2022-02-24 08:50:04 UTC
Created attachment 17178 [details]
patch for 4.15
Comment 12 Stefan Metzmacher 2022-02-24 09:47:57 UTC
Don't we want this in 4.14 too?
Comment 13 Samuel Cabrero 2022-02-24 09:58:20 UTC
(In reply to Stefan Metzmacher from comment #12)
Probably 4.13 too, enterprise principals were introduced in 4.12. I will do the backports.
Comment 14 Stefan Metzmacher 2022-02-24 10:08:09 UTC
(In reply to Samuel Cabrero from comment #13)

Thanks, but only 4.14 will get it applied, 4.13 is in security mode already
Comment 15 Samuel Cabrero 2022-02-24 10:26:46 UTC
Created attachment 17179 [details]
patch for 4.14
Comment 16 Jule Anger 2022-02-24 13:55:59 UTC
Pushed to autobuild-v4-{16,15,14}-test.
Comment 17 Samba QA Contact 2022-02-25 11:37:05 UTC
This bug was referenced in samba v4-14-test:

Comment 18 Samba QA Contact 2022-02-25 18:09:03 UTC
This bug was referenced in samba v4-16-test:

Comment 19 Samba QA Contact 2022-02-27 10:32:03 UTC
This bug was referenced in samba v4-15-test:

Comment 20 Jule Anger 2022-02-27 16:20:15 UTC
Closing out bug report.

Comment 21 Samba QA Contact 2022-03-01 08:55:13 UTC
This bug was referenced in samba v4-16-stable (Release samba-4.16.0rc4):

Comment 22 Samba QA Contact 2022-03-15 13:26:23 UTC
This bug was referenced in samba v4-15-stable (Release samba-4.15.6):

Comment 23 Samba QA Contact 2022-04-04 12:51:23 UTC
This bug was referenced in samba v4-14-stable (Release samba-4.14.13):

Comment 24 Samba QA Contact 2022-07-12 12:39:04 UTC
This bug was referenced in samba master:

Comment 25 Samuel Cabrero 2022-07-12 15:28:58 UTC
The issue was not completely fixed by the released patches and winbind still fails to refresh the tickets in certain scenarios or AD topologies. Details in https://gitlab.com/samba-team/samba/-/merge_requests/2610.
Comment 26 Samuel Cabrero 2022-07-12 15:39:39 UTC
Created attachment 17414 [details]
patch for 4.16 (part2)
Comment 27 Samuel Cabrero 2022-07-12 15:40:28 UTC
Created attachment 17415 [details]
patch for 4.15 (part2)
Comment 28 Samuel Cabrero 2022-07-12 15:40:59 UTC
Created attachment 17416 [details]
patch for 4.14 (part2)
Comment 29 Jule Anger 2022-07-18 07:56:11 UTC
Pushed to autobuild-v4-{16,15}-test.
4.14 is in security mode already.
Comment 30 Samba QA Contact 2022-07-18 09:41:04 UTC
This bug was referenced in samba v4-16-test:

Comment 31 Samba QA Contact 2022-07-18 10:37:03 UTC
This bug was referenced in samba v4-15-test:

Comment 32 Samba QA Contact 2022-07-18 11:17:52 UTC
This bug was referenced in samba v4-16-stable (Release samba-4.16.3):

Comment 33 Jule Anger 2022-07-31 18:04:11 UTC
Closing out bug report.

Comment 34 Samba QA Contact 2022-09-28 15:40:35 UTC
This bug was referenced in samba v4-15-stable (Release samba-4.15.10):