The scenario for my customers: - There are a lot of SLES 11 SP3 servers in the customers environment. - The SLES servers are configured to authenticate AD users using Kerberos. - Winbind has been configured to cache successful Kerberos authentications. - This is configured and working. - It will continue to work in their environments until they change a working users password in AD. - That change appears to be ok at first, but eventually that users account in AD will get locked. - When that happens the Windows AD servers log files will blame one of the SLES servers for attempting too many invalid authentication attempts. - To the AD users knowledge they may not have even accessed that SLES server that day with that user. So here's what's happening: Say USER_A is the problem user. 'winbind refresh tickets' is enabled on a machine to which USER_A has authenticated at some point. The machine is put to sleep, or in some other way is disconnected from AD for a period of time. Due to the period of disconnect, winbind cannot renew USER_A's tickets. When the machine comes back online, winbind uses USER_A's cached credentials to authenticate (see krb5_ticket_refresh_handler()). In *most* situations, this works as intended, and USER_A will regain his tickets. By most situations, I mean, the password has not changed, OR (here's the tricky part) the domain controller is 2003 server (with all updates installed) or above and the password hasn't been changed more than twice, because MS server ignores bad password attempts if committed using a recent (previous 2) password. The failure is when the auto kinit uses the wrong password, and either we're authenticating to an old server, or the password has been changed more than twice recently. In this case, MS server logs a badPwdCount. Now, a single bad password wont lock the user out, but it only takes <max bad passwd count> number of computers, all coming back online at the same moment, and the user is locked out immediately. So, previously auth'd computers going down, then coming up after the user's tickets have expired, cause the lockout. This was originally reported in samba 3.6.3, but I was able to reproduce this on today's samba master. <patchset to follow>
Created attachment 13903 [details] patch that fixes the bad password attempts
Comment on attachment 13903 [details] patch that fixes the bad password attempts Nice work David ! Can you change the DEBUG(10) call to a more modern DBG_NOTICE() call please, and also log the username that this happened to. This will help admins looking into this condition. Also, use strtoull() to convert the uint64_t value instead of atoll() (yeah I know the old ads_pull_uint32() uses atoi, but that's old and should be modernized too :-). Thanks, Jeremy.
Created attachment 13904 [details] patch that fixes the bad password attempts
*** Bug 12443 has been marked as a duplicate of this bug. ***
The strategy implemented with the patch here is rather different than the one on the pull request (https://github.com/samba-team/samba/pull/69). To my untrained eye, it seems like both might be worthwhile? The patch here tries to magic detection of whether this kinit-redo is safe where the other just disables it wholesale. Side note: Putting machines to sleep isn't necessary. I've found that, if I ever authenticate on a machine via password & winbind, it will continue trying to renew the ticket indefinitely, even long after I've logged out and any ssh sessions have terminated. This causes accounts to get locked out any time a password is changed, if the user in question was logged into more than 3 other machines besides the one from which they changed their password.
Just a comment: Is it ever a good idea to use the cached plain text password to re-kinit? Shouldn't we rely on reneable tickets? There was a reason why AD has a limited ticket lifetime. To me it seems like "winbind refresh tickets" using plain text credentials should be only for long-running, cronjob-like service accounts, and not by default for normal interactive users. Wouldn't "winbind refresh tickets" being changed to a list of usernames or possibly something similar to the pam required membership sid thingy help more? For anybody not in this list we just do the standard ticket renewal using plain kerberos mechanisms without cheating?
(In reply to Matthew "Cheetah" Gabeler-Lee from comment #5) You missed part of the conversation on the mailing list (and I forgot that pull request was sitting there). My initial reaction to this bug, is that we *should not* be using offline creds to kinit when renewal has failed. Hence the original patch, which provides it as an option, but disables it by default. After getting no response for that patch, I wrote the new patch (which keeps this behavior, but checks whether the password has changed to avoid locking out a user). I still think the right approach is to simply remove the password kinit, although Volker makes a good point about long running services. Maybe the best patch would be a combination of, making "winbind password kinit" a list of users to perform a kinit for if creds expire, while also including the patch that doesn't lock the user out if this does happen to be enabled.
(In reply to David Mulder from comment #7) Yes, having "winbind password kinit" as a list with support for "*" would be good. The only question is how are users specified, by UPN or DOMAIN\user.
(In reply to Stefan Metzmacher from comment #8) And "winbind password kinit" should default to * for one release, then be disabled in the following release.
(In reply to Stefan Metzmacher from comment #8) The only problem with this approach, is if we include the patch for checking the last password change, we'll end up having ldap calls within winbind still (which you mentioned we're trying to move away from).
(In reply to David Mulder from comment #10) Even with the default of "*", I wouldn't do ldap calls. If someone hits that bug, the admin can disable this or only configure it for account which will never change the password.
(In reply to Stefan Metzmacher from comment #11) We have the refresh with password code over 10 years already and it took several years to get the first bug report, so I guess not many people get this at all. And if they get it they likely use a samba version without the fix, when they got it and have the fix they can also change their config file.
Created attachment 14715 [details] patch which creates a list of UPNs to renew via password
Hi David, can you describe the problem more detailed again? I recently found out that Windows DCs (tested with 2018R2) don't update the the badPwdCount is the password is included in the old or older keys of package_PrimaryKerberosCtr4. What kind of DC where you using? Was the password changed multiple times? I guess we need to fix our KDC to check old and older keys, see bug #14054. And for winbindd we should remember if kinit failed once and never retry with that password again.
(In reply to Stefan Metzmacher from comment #14) FYI, the patch for this bug was never merged. Still needs two approvers. The bug you mentioned is exactly what this bug is referring to. I believe the server behavior of not incrementing the badPwdCount for the previous 2 bad passwords was introduced in a service pack to 2003, IIRC. The customers we were working with were using 2003. See here for more info: https://social.technet.microsoft.com/wiki/contents/articles/32490.active-directory-bad-passwords-and-account-lockout.aspx#Account_Lockout_in_Windows_TwoThousandThree_and_Above In this situation, the password only needed to be changed once, before there were lockouts, since they were using an older windows server. This is probably the same situation people are now encountering in the samba dc.