Hi I described our problem here but was not able to resolf it and I think it is a bug: https://lists.samba.org/archive/samba/2025-February/251035.html But to be sure here is a up to date summary and sorry for the long bug report: If I run net ads changetrustpw I get this: Changing password for principal: host$@DOMAIN.COM gensec_gse_client_prepare_ccache: Kinit for HOST$@DOMAIN.COM to access ldap/dc2.domain.com failed: Preauthentication failed: NT_STATUS_LOGON_FAILURE pw2kt_get_dc_info: Failed to refresh keytab, ads_connect() returned Invalid credentials secrets_finish_password_change: Sync of machine password failed. Password change failed: An internal error occurred. The keytab is still updated with the new KVNO and the machine password in AD is updated. However the new KVNO is appended to the keytab. So there are now 4 KVNOs in the keytab. Normally there are only 3. It does not matter if I set any TLS options or not. This is different to the thread in the mailing list. I did some more tests and it does not matter. It is just that this does not happen all the time. Sometimes it looks like this: net ads changetrustpw Changing password for principal: host$@DOMAIN.COM gensec_gse_client_prepare_ccache: Kinit for host$@DOMAIN.COM to access ldap/dc4.domain.com failed: Preauthentication failed: NT_STATUS_LOGON_FAILURE Password change for principal host$@DOMAIN.COM succeeded. Then the keytab is upated and only the 3 last KVNOs are present. If I change the password with this command: wbinfo --change-secret --domain=DOMAIN I never get an error. However, if I set the password expiration to the default or any other value other then 0 I sometimes can not log in via Kerberos as the keytab is not updated. I tried disabling the automatic password change and do it via a cronjob. This works for some time and then the keytab is again not updated. If I run "net ads changetrustpw" I see on the DC side mixed entries in the logs: One dc2 a [Success]. Feb 19 11:27:22 dc2.domain.com samba[8970]: Password Change [Change] at [Wed, 19 Feb 2025 11:27:22.744358 CET] status [Success] remote host [Unknown] SID [S-1-5-21-xx-xx-xx-xx] DN [CN=HOST,CN=Computers,DC=domain,DC=com] On dc4 [insufficient access rights] Password Change [Reset] at [Wed, 19 Feb 2025 11:27:22.667348 CET] status [insufficient access rights] remote host [Unknown] SID [S-1-5-21-xx-xx-xx-xx] DN [CN=HOST,CN=Computers,DC=domain,DC=com] Feb 19 11:27:22 dc4.domain.com samba[4078]: [2025/02/19 11:27:22.667406, 5] ../../lib/audit_logging/audit_logging.c:97(audit_log_human_text) Feb 19 11:27:22 dc4.domain.com samba[4078]: DSDB Transaction [rollback] at [Wed, 19 Feb 2025 11:27:22.667402 CET] duration [1558] Feb 19 11:27:22 dc4.domain.com samba[4078]: [2025/02/19 11:27:22.667485, 0] ../../source4/kdc/kpasswd-service-heimdal.c:234(kpasswd_set_password) Feb 19 11:27:22 dc4.domain.com samba[4078]: kpasswd_set_password: kpasswd_samdb_set_password failed - NT_STATUS_ACCESS_DENIED If the password change is done because of "machine password timeout" then it looks like this on the DC: Feb 19 11:37:22 dc2.domain.com samba[8914]: Password Change [Reset] at [Wed, 19 Feb 2025 11:37:22.503303 CET] status [Success] remote host [ipv4:192.168.0.31:55402] SID [S-1-5-18] DN [CN=HOST,CN=Computers,DC=domain,DC=com] Feb 19 11:37:22 dc2.domain.com samba[8978]: [2025/02/19 11:37:22.639002, 2] ../../auth/auth_log.c:876(log_authentication_event_human_readable) No logs in the other DCs. Locally it looks like this: Feb 19 11:37:22 host.domain.com winbindd[31776]: [2025/02/19 11:37:22.434123, 0, traceid=1] ../../source3/libads/trusts_util.c:399(trust_pw_change) Feb 19 11:37:22 host.domain.com winbindd[31776]: 2025/02/19 11:37:22 : trust_pw_change(DOMAIN): Verifying passwords remotely netlogon_creds_cli:CLI[host/host$]/SRV[DC2/DOMAIN]. Feb 19 11:37:22 host.domain.com winbindd[31776]: [2025/02/19 11:37:22.438574, 0, traceid=1] ../../source3/libads/trusts_util.c:477(trust_pw_change) Feb 19 11:37:22 host.domain.com winbindd[31776]: 2025/02/19 11:37:22 : trust_pw_change(DOMAIN): Verified old password remotely using netlogon_creds_cli:CLI[host/host$]/SRV[DC2/DOMAIN] Feb 19 11:37:22 host.domain.com winbindd[31776]: [2025/02/19 11:37:22.438683, 0, traceid=1] ../../source3/libads/trusts_util.c:516(trust_pw_change) Feb 19 11:37:22 host.domain.com winbindd[31776]: 2025/02/19 11:37:22 : trust_pw_change(DOMAIN): Changed password locally Feb 19 11:37:22 host.domain.com winbindd[31776]: [2025/02/19 11:37:22.510568, 0, traceid=1] ../../source3/libads/trusts_util.c:570(trust_pw_change) Feb 19 11:37:22 host.domain.com winbindd[31776]: 2025/02/19 11:37:22 : trust_pw_change(DOMAIN): Changed password remotely using netlogon_creds_cli:CLI[host/host$]/SRV[DC2/DOMAIN] Feb 19 11:37:22 host.domain.com winbindd[31776]: [2025/02/19 11:37:22.511555, 1, traceid=1] ../../source3/passdb/machine_account_secrets.c:786(secrets_debug_domain_info) Feb 19 11:37:22 host.domain.com winbindd[31776]: &sdib: struct secrets_domain_infoB Feb 19 11:37:22 host.domain.com winbindd[31776]: version : SECRETS_DOMAIN_INFO_VERSION_1 (1) Feb 19 11:37:22 host.domain.com winbindd[31776]: reserved : 0x00000000 (0) Feb 19 11:37:22 host.domain.com winbindd[31776]: info : union secrets_domain_infoU(case 1) Feb 19 11:37:22 host.domain.com winbindd[31776]: info1 : * Feb 19 11:37:22 host.domain.com winbindd[31776]: info1: struct secrets_domain_info1 Feb 19 11:37:22 host.domain.com winbindd[31776]: reserved_flags : 0x0000000000000000 (0) Feb 19 11:37:22 host.domain.com winbindd[31776]: join_time : Mon Feb 17 16:20:16 2025 CET Feb 19 11:37:22 host.domain.com winbindd[31776]: computer_name : 'host' Feb 19 11:37:22 host.domain.com winbindd[31776]: account_name : 'host$' Feb 19 11:37:22 host.domain.com winbindd[31776]: secure_channel_type : SEC_CHAN_WKSTA (2) Feb 19 11:37:22 host.domain.com winbindd[31776]: domain_info: struct lsa_DnsDomainInfo Feb 19 11:37:22 host.domain.com winbindd[31776]: name: struct lsa_StringLarge Feb 19 11:37:22 host.domain.com winbindd[31776]: length : 0x0010 (16) Feb 19 11:37:22 host.domain.com winbindd[31776]: size : 0x0012 (18) Feb 19 11:37:22 host.domain.com winbindd[31776]: string : * Feb 19 11:37:22 host.domain.com winbindd[31776]: string : 'DOMAIN' Feb 19 11:37:22 host.domain.com winbindd[31776]: dns_domain: struct lsa_StringLarge Feb 19 11:37:22 host.domain.com winbindd[31776]: length : 0x0026 (38) Feb 19 11:37:22 host.domain.com winbindd[31776]: size : 0x0028 (40) Feb 19 11:37:22 host.domain.com winbindd[31776]: string : * Feb 19 11:37:22 host.domain.com winbindd[31776]: string : 'domain.com' Feb 19 11:37:22 host.domain.com winbindd[31776]: dns_forest: struct lsa_StringLarge Feb 19 11:37:22 host.domain.com winbindd[31776]: length : 0x0026 (38) Feb 19 11:37:22 host.domain.com winbindd[31776]: size : 0x0028 (40) Feb 19 11:37:22 host.domain.com winbindd[31776]: string : * Feb 19 11:37:22 host.domain.com winbindd[31776]: string : 'domain.com' Feb 19 11:37:22 host.domain.com winbindd[31776]: domain_guid : 733e196a-bcc5-407f-8de5-76e577927c13 Feb 19 11:37:22 host.domain.com winbindd[31776]: sid : * Feb 19 11:37:22 host.domain.com winbindd[31776]: sid : S-1-5-21-773202902-494389186-2375354597 Feb 19 11:37:22 host.domain.com winbindd[31776]: trust_flags : 0x0000001a (26) Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: NETR_TRUST_FLAG_IN_FOREST Feb 19 11:37:22 host.domain.com winbindd[31776]: 1: NETR_TRUST_FLAG_OUTBOUND Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: NETR_TRUST_FLAG_TREEROOT Feb 19 11:37:22 host.domain.com winbindd[31776]: 1: NETR_TRUST_FLAG_PRIMARY Feb 19 11:37:22 host.domain.com winbindd[31776]: 1: NETR_TRUST_FLAG_NATIVE Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: NETR_TRUST_FLAG_INBOUND Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: NETR_TRUST_FLAG_MIT_KRB5 Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: NETR_TRUST_FLAG_AES Feb 19 11:37:22 host.domain.com winbindd[31776]: trust_type : LSA_TRUST_TYPE_UPLEVEL (2) Feb 19 11:37:22 host.domain.com winbindd[31776]: trust_attributes : 0x00000040 (64) Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: LSA_TRUST_ATTRIBUTE_NON_TRANSITIVE Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: LSA_TRUST_ATTRIBUTE_UPLEVEL_ONLY Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: LSA_TRUST_ATTRIBUTE_QUARANTINED_DOMAIN Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: LSA_TRUST_ATTRIBUTE_FOREST_TRANSITIVE Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: LSA_TRUST_ATTRIBUTE_CROSS_ORGANIZATION Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: LSA_TRUST_ATTRIBUTE_WITHIN_FOREST Feb 19 11:37:22 host.domain.com winbindd[31776]: 1: LSA_TRUST_ATTRIBUTE_TREAT_AS_EXTERNAL Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: LSA_TRUST_ATTRIBUTE_USES_RC4_ENCRYPTION Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: LSA_TRUST_ATTRIBUTE_CROSS_ORGANIZATION_NO_TGT_DELEGATION Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: LSA_TRUST_ATTRIBUTE_PIM_TRUST Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: LSA_TRUST_ATTRIBUTE_CROSS_ORGANIZATION_ENABLE_TGT_DELEGATION Feb 19 11:37:22 host.domain.com winbindd[31776]: reserved_routing : NULL Feb 19 11:37:22 host.domain.com winbindd[31776]: supported_enc_types : 0x0000001c (28) Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: KERB_ENCTYPE_DES_CBC_CRC Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: KERB_ENCTYPE_DES_CBC_MD5 Feb 19 11:37:22 host.domain.com winbindd[31776]: 1: KERB_ENCTYPE_RC4_HMAC_MD5 Feb 19 11:37:22 host.domain.com winbindd[31776]: 1: KERB_ENCTYPE_AES128_CTS_HMAC_SHA1_96 Feb 19 11:37:22 host.domain.com winbindd[31776]: 1: KERB_ENCTYPE_AES256_CTS_HMAC_SHA1_96 Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: KERB_ENCTYPE_AES256_CTS_HMAC_SHA1_96_SK Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: KERB_ENCTYPE_FAST_SUPPORTED Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: KERB_ENCTYPE_COMPOUND_IDENTITY_SUPPORTED Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: KERB_ENCTYPE_CLAIMS_SUPPORTED Feb 19 11:37:22 host.domain.com winbindd[31776]: 0: KERB_ENCTYPE_RESOURCE_SID_COMPRESSION_DISABLED Feb 19 11:37:22 host.domain.com winbindd[31776]: salt_principal : * Feb 19 11:37:22 host.domain.com winbindd[31776]: salt_principal : 'host/host.domain.com at domain.com' Feb 19 11:37:22 host.domain.com winbindd[31776]: password_last_change : Wed Feb 19 11:37:22 2025 CET Feb 19 11:37:22 host.domain.com winbindd[31776]: password_changes : 0x0000000000000028 (40) Feb 19 11:37:22 host.domain.com winbindd[31776]: next_change : NULL Feb 19 11:37:22 host.domain.com winbindd[31776]: password : * Feb 19 11:37:22 host.domain.com winbindd[31776]: password: struct secrets_domain_info1_password Feb 19 11:37:22 host.domain.com winbindd[31776]: change_time : Wed Feb 19 11:37:22 2025 CET Feb 19 11:37:22 host.domain.com winbindd[31776]: change_server : 'dc2.domain.com' Feb 19 11:37:22 host.domain.com winbindd[31776]: cleartext_blob : DATA_BLOB length=240 Feb 19 11:37:22 host.domain.com winbindd[31776]: nt_hash: struct samr_Password Feb 19 11:37:22 host.domain.com winbindd[31776]: hash: ARRAY(16): <REDACTED SECRET VALUES> Feb 19 11:37:22 host.domain.com winbindd[31776]: salt_data : * Feb 19 11:37:22 host.domain.com winbindd[31776]: salt_data : 'domain.comhosthost.domain.com' Feb 19 11:37:22 host.domain.com winbindd[31776]: default_iteration_count : 0x00001000 (4096) Feb 19 11:37:22 host.domain.com winbindd[31776]: num_keys : 0x0003 (3) Feb 19 11:37:22 host.domain.com winbindd[31776]: keys: ARRAY(3) Feb 19 11:37:22 host.domain.com winbindd[31776]: keys: struct secrets_domain_info1_kerberos_key Feb 19 11:37:22 host.domain.com winbindd[31776]: keytype : 0x00000012 (18) Feb 19 11:37:22 host.domain.com winbindd[31776]: iteration_count : 0x00001000 (4096) Feb 19 11:37:22 host.domain.com winbindd[31776]: value : DATA_BLOB length=32 Feb 19 11:37:22 host.domain.com winbindd[31776]: keys: struct secrets_domain_info1_kerberos_key Feb 19 11:37:22 host.domain.com winbindd[31776]: keytype : 0x00000011 (17) Feb 19 11:37:22 host.domain.com winbindd[31776]: iteration_count : 0x00001000 (4096) Feb 19 11:37:22 host.domain.com winbindd[31776]: value : DATA_BLOB length=16 Feb 19 11:37:22 host.domain.com winbindd[31776]: keys: struct secrets_domain_info1_kerberos_key Feb 19 11:37:22 host.domain.com winbindd[31776]: keytype : 0x00000017 (23) Feb 19 11:37:22 host.domain.com winbindd[31776]: iteration_count : 0x00001000 (4096) Feb 19 11:37:22 host.domain.com winbindd[31776]: value : DATA_BLOB length=16 Feb 19 11:37:22 host.domain.com winbindd[31776]: old_password : * Feb 19 11:37:22 host.domain.com winbindd[31776]: old_password: struct secrets_domain_info1_password Feb 19 11:37:22 host.domain.com winbindd[31776]: change_time : Wed Feb 19 11:27:22 2025 CET Feb 19 11:37:22 host.domain.com winbindd[31776]: change_server : '192.168.0.91' Feb 19 11:37:22 host.domain.com winbindd[31776]: cleartext_blob : DATA_BLOB length=240 Feb 19 11:37:22 host.domain.com winbindd[31776]: nt_hash: struct samr_Password Feb 19 11:37:22 host.domain.com winbindd[31776]: hash: ARRAY(16): <REDACTED SECRET VALUES> Feb 19 11:37:22 host.domain.com winbindd[31776]: salt_data : * Feb 19 11:37:22 host.domain.com winbindd[31776]: salt_data : 'domain.comhosthost.domain.com' Feb 19 11:37:22 host.domain.com winbindd[31776]: default_iteration_count : 0x00001000 (4096) Feb 19 11:37:22 host.domain.com winbindd[31776]: num_keys : 0x0003 (3) Feb 19 11:37:22 host.domain.com winbindd[31776]: keys: ARRAY(3) Feb 19 11:37:22 host.domain.com winbindd[31776]: keys: struct secrets_domain_info1_kerberos_key Feb 19 11:37:22 host.domain.com winbindd[31776]: keytype : 0x00000012 (18) Feb 19 11:37:22 host.domain.com winbindd[31776]: iteration_count : 0x00001000 (4096) Feb 19 11:37:22 host.domain.com winbindd[31776]: value : DATA_BLOB length=32 Feb 19 11:37:22 host.domain.com winbindd[31776]: keys: struct secrets_domain_info1_kerberos_key Feb 19 11:37:22 host.domain.com winbindd[31776]: keytype : 0x00000011 (17) Feb 19 11:37:22 host.domain.com winbindd[31776]: iteration_count : 0x00001000 (4096) Feb 19 11:37:22 host.domain.com winbindd[31776]: value : DATA_BLOB length=16 Feb 19 11:37:22 host.domain.com winbindd[31776]: keys: struct secrets_domain_info1_kerberos_key Feb 19 11:37:22 host.domain.com winbindd[31776]: keytype : 0x00000017 (23) Feb 19 11:37:22 host.domain.com winbindd[31776]: iteration_count : 0x00001000 (4096) Feb 19 11:37:22 host.domain.com winbindd[31776]: value : DATA_BLOB length=16 Feb 19 11:37:22 host.domain.com winbindd[31776]: older_password : * Feb 19 11:37:22 host.domain.com winbindd[31776]: older_password: struct secrets_domain_info1_password Feb 19 11:37:22 host.domain.com winbindd[31776]: change_time : Wed Feb 19 11:26:01 2025 CET Feb 19 11:37:22 host.domain.com winbindd[31776]: change_server : 'dc1' Feb 19 11:37:22 host.domain.com winbindd[31776]: cleartext_blob : DATA_BLOB length=240 Feb 19 11:37:22 host.domain.com winbindd[31776]: nt_hash: struct samr_Password Feb 19 11:37:22 host.domain.com winbindd[31776]: hash: ARRAY(16): <REDACTED SECRET VALUES> Feb 19 11:37:22 host.domain.com winbindd[31776]: salt_data : * Feb 19 11:37:22 host.domain.com winbindd[31776]: salt_data : 'domain.comhosthost.domain.com' Feb 19 11:37:22 host.domain.com winbindd[31776]: default_iteration_count : 0x00001000 (4096) Feb 19 11:37:22 host.domain.com winbindd[31776]: num_keys : 0x0003 (3) Feb 19 11:37:22 host.domain.com winbindd[31776]: keys: ARRAY(3) Feb 19 11:37:22 host.domain.com winbindd[31776]: keys: struct secrets_domain_info1_kerberos_key Feb 19 11:37:22 host.domain.com winbindd[31776]: keytype : 0x00000012 (18) Feb 19 11:37:22 host.domain.com winbindd[31776]: iteration_count : 0x00001000 (4096) Feb 19 11:37:22 host.domain.com winbindd[31776]: value : DATA_BLOB length=32 Feb 19 11:37:22 host.domain.com winbindd[31776]: keys: struct secrets_domain_info1_kerberos_key Feb 19 11:37:22 host.domain.com winbindd[31776]: keytype : 0x00000011 (17) Feb 19 11:37:22 host.domain.com winbindd[31776]: iteration_count : 0x00001000 (4096) Feb 19 11:37:22 host.domain.com winbindd[31776]: value : DATA_BLOB length=16 Feb 19 11:37:22 host.domain.com winbindd[31776]: keys: struct secrets_domain_info1_kerberos_key Feb 19 11:37:22 host.domain.com winbindd[31776]: keytype : 0x00000017 (23) Feb 19 11:37:22 host.domain.com winbindd[31776]: iteration_count : 0x00001000 (4096) Feb 19 11:37:22 host.domain.com winbindd[31776]: value : DATA_BLOB length=16 Feb 19 11:37:22 host.domain.com winbindd[31776]: [2025/02/19 11:37:22.716105, 0, traceid=1] ../../source3/libads/trusts_util.c:594(trust_pw_change) Feb 19 11:37:22 host.domain.com winbindd[31776]: 2025/02/19 11:37:22 : trust_pw_change(DOMAIN): Finished password change. Feb 19 11:37:22 host.domain.com winbindd[31776]: [2025/02/19 11:37:22.721540, 0, traceid=1] ../../source3/libads/trusts_util.c:646(trust_pw_change) Feb 19 11:37:22 host.domain.com winbindd[31776]: 2025/02/19 11:37:22 : trust_pw_change(DOMAIN): Verified new password remotely using netlogon_creds_cli:CLI[host/host$]/SRV[DC2/DOMAIN] I suspect it has something to do with the multiple DCs but I can not pinpoint it. One more thing to point out is if I set "dedicated keytab file = /etc/krb5.keytab" in smb.con as we used to, automatic password change is disabled. Which I never found documented. Current versions are: 4.22.1-SerNet-RedHat-5.el8 on the domain member (also tried 4.21.5) 4.21.5-SerNet-RedHat-6.el8 on the 4 DCs Same problem on a Debian system with 4.21.3-Debian-4.21.3+dfsg-6~~mjt+deb12 as domain member Here is the current smb.conf [global] netbios name = HOST server string = Daten security = ADS realm = DOMAIN.COM workgroup = BRAIN-02 disable netbios = yes smb ports = 445 interfaces = eth0 bind interfaces only = yes server min protocol = SMB2 client min protocol = SMB2 log level = 1 auth_audit:5 client ldap sasl wrapping = starttls tls cafile = tls/ca.pem tls verify peer = ca_and_name tls crlfile = tls/root.crl.pem logging = syslog only sync machine password to keytab = /etc/krb5.keytab:sync_spns:sync_kvno:machine_password writeable =YES map acl inherit = yes store dos attributes = yes inherit acls = Yes vfs objects = acl_xattr full_audit full_audit:success = pwrite write unlinkat renameat full_audit:prefix = IP=%I|USER=%u|MACHINE=%m|VOLUME=%S full_audit:priority = NOTICE full_audit:facility = local7 full_audit:failure = none username map = /etc/samba/smbusers interfaces = lo eth0 bind interfaces only = Yes ##idmap## # Default idmap config used for BUILTIN and local windows accounts/groups idmap config *:backend = tdb idmap config *:range = 1000000-2000000 # idmap config for domain DOMAIN idmap config DOMAIN:backend = ad idmap config DOMAIN:range = 500-65555 idmap config DOMAIN:unix_nss_info = yes idmap config DOMAIN:schema_mode = rfc2307 winbind enum users = yes winbind enum groups = yes winbind use default domain = Yes #machine password timeout = 604800 machine password timeout = 0 #winbind offline logon = yes winbind reconnect delay = 5 winbind refresh tickets = yes min domain uid = 500