13336 – unable to re-join 4.8.0 controller to 4.7.5 domain

Bug 13336 - unable to re-join 4.8.0 controller to 4.7.5 domain

Summary: unable to re-join 4.8.0 controller to 4.7.5 domain

Status:	RESOLVED FIXED

Alias:	None

Product:	Samba 4.1 and newer
Classification:	Unclassified
Component:	AD: LDB/DSDB/SAMDB (show other bugs)
Version:	4.8.0
Hardware:	All Linux

Importance:	P5 critical (vote)
Target Milestone:	---
Assignee:	Andrew Bartlett
QA Contact:	Samba QA Contact

URL:
Keywords:

Depends on:
Blocks:

Reported:	2018-03-15 16:23 UTC by Alexey Vekshin
Modified:	2018-06-07 16:03 UTC (History)
CC List:	0 users

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Alexey Vekshin 2018-03-15 16:23:42 UTC

After failed in-place upgrade from 4.7.5 I've wiped /var/lib/samba/private and attempted to re-join controller to domain (using healthy 4.7.5 DC); sadly, join fails, first bad sign being "partition_metadata: Migrating partition metadata: open of metadata.tdb gave: (null)" below:

================================================================
root@dc1-u1-vm:~# samba-tool domain join ad.maxidom.ru DC --username joiner --server=ad-pdc.ad.maxidom.ru --dns-backend=SAMBA_INTERNAL --option="bind interfaces only=yes" --option="interfaces=lo eth0"
Password for [AD\joiner]:
workgroup is AD
realm is ad.maxidom.ru
Adding CN=DC1-U1,OU=Domain Controllers,DC=ad,DC=maxidom,DC=ru
Adding CN=DC1-U1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ad,DC=maxidom,DC=ru
Adding CN=NTDS Settings,CN=DC1-U1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ad,DC=maxidom,DC=ru
Adding SPNs to CN=DC1-U1,OU=Domain Controllers,DC=ad,DC=maxidom,DC=ru
Setting account password for DC1-U1$
Enabling account
Calling bare provision
lpcfg_load: refreshing parameters from /etc/samba/smb.conf
Looking up IPv4 addresses
Looking up IPv6 addresses
No IPv6 address will be assigned
Setting up share.ldb
Setting up secrets.ldb
Setting up the registry
key added: key=SOFTWARE,hive=NONE
key added: key=Microsoft,key=SOFTWARE,hive=NONE
key added: key=Windows NT,key=Microsoft,key=SOFTWARE,hive=NONE
key added: key=CurrentVersion,key=Windows NT,key=Microsoft,key=SOFTWARE,hive=NONE
key added: key=SYSTEM,hive=NONE
key added: key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Control,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=ProductOptions,key=Control,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Print,key=Control,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Terminal Server,key=Control,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Services,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Netlogon,key=Services,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Parameters,key=Netlogon,key=Services,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Alerter,key=Services,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Parameters,key=Alerter,key=Services,key=CurrentControlSet,key=SYSTEM,hive=NONE
Setting up the privileges database
Setting up idmap db
Setting up SAM db
Setting up sam.ldb partitions and settings
Setting up sam.ldb rootDSE
Pre-loading the Samba 4 and AD schema
partition_metadata: Migrating partition metadata: open of metadata.tdb gave: (null)
Unable to determine the DomainSID, can not enforce uniqueness constraint on local domainSIDs

A Kerberos configuration suitable for Samba AD has been generated at /var/lib/samba/private/krb5.conf
Merge the contents of this file with your system krb5.conf or replace it with this one. Do not create a symlink!
Provision OK for domain DN DC=ad,DC=maxidom,DC=ru
Starting replication
Schema-DN[CN=Schema,CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[402/1550] linked_values[0/0]
Schema-DN[CN=Schema,CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[804/1550] linked_values[0/0]
Schema-DN[CN=Schema,CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[1206/1550] linked_values[0/0]
Schema-DN[CN=Schema,CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[1550/1550] linked_values[0/0]
Analyze and apply schema objects
Partition[CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[402/1643] linked_values[0/1]
Partition[CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[804/1643] linked_values[0/1]
Partition[CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[1206/1643] linked_values[0/1]
Partition[CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[1608/1643] linked_values[0/2]
Partition[CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[1643/1643] linked_values[66/66]
Replicating critical objects from the base DN of the domain
Partition[DC=ad,DC=maxidom,DC=ru] objects[100/100] linked_values[29/29]
Partition[DC=ad,DC=maxidom,DC=ru] objects[503/512] linked_values[0/30]
Failed to apply records: Failed to locally apply remote add of CN=WDS,OU=servers,OU=all-computers,DC=ad,DC=maxidom,DC=ru: ../ldb_tdb/ldb_index.c:2012: Failed to re-index servicePrincipalName in CN=WDS,OU=servers,OU=all-computers,DC=ad,DC=maxidom,DC=ru - (null): Operations error
Failed to commit objects: WERR_GEN_FAILURE
Join failed - cleaning up
Could not find machine account in secrets database: Failed to fetch machine account password for AD from both secrets.ldb (Could not find entry to match filter: '(&(flatname=AD)(objectclass=primaryDomain))' base: 'cn=Primary Domains': No such object: dsdb_search at ../source4/dsdb/common/util.c:4641) and from /var/lib/samba/private/secrets.tdb: NT_STATUS_CANT_ACCESS_DOMAIN_INFO
Deleted CN=DC1-U1,OU=Domain Controllers,DC=ad,DC=maxidom,DC=ru
Deleted CN=NTDS Settings,CN=DC1-U1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ad,DC=maxidom,DC=ru
Deleted CN=DC1-U1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ad,DC=maxidom,DC=ru
ERROR(runtime): uncaught exception - (31, "Failed to process 'chunk' of DRS replicated objects: WERR_GEN_FAILURE")
  File "/usr/lib64/python2.6/site-packages/samba/netcmd/__init__.py", line 176, in _run
    return self.run(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/samba/netcmd/domain.py", line 706, in run
    plaintext_secrets=plaintext_secrets)
  File "/usr/lib64/python2.6/site-packages/samba/join.py", line 1482, in join_DC
    ctx.do_join()
  File "/usr/lib64/python2.6/site-packages/samba/join.py", line 1383, in do_join
    ctx.join_replicate()
  File "/usr/lib64/python2.6/site-packages/samba/join.py", line 942, in join_replicate
    replica_flags=ctx.domain_replica_flags)
  File "/usr/lib64/python2.6/site-packages/samba/drs_utils.py", line 330, in replicate
    raise e
================================================================

Comment 1 Alexey Vekshin 2018-03-15 16:51:12 UTC

record in question looks rather ordinary; it is windows 2012 member server, used as WDS:

================================================================
root@ad-bdc-vm:alex# ldbsearch -H /var/lib/samba/private/sam.ldb -b 'CN=WDS,OU=servers,OU=all-computers,DC=ad,DC=maxidom,DC=ru' servicePrincipalName
# record 1
dn: CN=WDS,OU=servers,OU=all-computers,DC=ad,DC=maxidom,DC=ru
servicePrincipalName: HOST/wds.ad.maxidom.ru
servicePrincipalName: RestrictedKrbHost/wds.ad.maxidom.ru
servicePrincipalName: HOST/WDS
servicePrincipalName: RestrictedKrbHost/WDS
servicePrincipalName: WSMAN/wds.ad.maxidom.ru
servicePrincipalName: WSMAN/wds
servicePrincipalName: TERMSRV/wds.ad.maxidom.ru
servicePrincipalName: TERMSRV/WDS
servicePrincipalName: TERMSRV/WDS.ad.maxidom.ru
================================================================

Comment 2 Alexey Vekshin 2018-03-15 18:42:12 UTC

component versions
- talloc 2.1.11
- tevent 0.9.36
- tdb    1.3.15
- ldb    1.3.2

installed from custom rpms, specs are mostly like SUSE with DC enabled (https://build.opensuse.org/project/show/network:samba:STABLE)

Comment 3 Alexey Vekshin 2018-03-16 06:51:17 UTC

Successful join log after downgrade to 4.7.6:

================================================================
root@dc1-u1-vm:alex# samba-tool domain join ad.maxidom.ru DC --username joiner --server=ad-pdc.ad.maxidom.ru --dns-backend=SAMBA_INTERNAL --option="bind interfaces only=yes" --option="interfaces=lo eth0"
Password for [AD\joiner]:
workgroup is AD
realm is ad.maxidom.ru
Adding CN=DC1-U1,OU=Domain Controllers,DC=ad,DC=maxidom,DC=ru
Adding CN=DC1-U1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ad,DC=maxidom,DC=ru
Adding CN=NTDS Settings,CN=DC1-U1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ad,DC=maxidom,DC=ru
Error reading smb_krb5 reply packet: NT_STATUS_CONNECTION_REFUSED from 10.84.128.10
Error reading smb_krb5 reply packet: NT_STATUS_CONNECTION_REFUSED from 10.84.128.10
Adding SPNs to CN=DC1-U1,OU=Domain Controllers,DC=ad,DC=maxidom,DC=ru
Setting account password for DC1-U1$
Enabling account
Calling bare provision
lpcfg_load: refreshing parameters from /etc/samba/smb.conf
Looking up IPv4 addresses
Looking up IPv6 addresses
No IPv6 address will be assigned
Setting up share.ldb
Setting up secrets.ldb
Setting up the registry
key added: key=SOFTWARE,hive=NONE
key added: key=Microsoft,key=SOFTWARE,hive=NONE
key added: key=Windows NT,key=Microsoft,key=SOFTWARE,hive=NONE
key added: key=CurrentVersion,key=Windows NT,key=Microsoft,key=SOFTWARE,hive=NONE
key added: key=SYSTEM,hive=NONE
key added: key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Control,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=ProductOptions,key=Control,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Print,key=Control,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Terminal Server,key=Control,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Services,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Netlogon,key=Services,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Parameters,key=Netlogon,key=Services,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Alerter,key=Services,key=CurrentControlSet,key=SYSTEM,hive=NONE
key added: key=Parameters,key=Alerter,key=Services,key=CurrentControlSet,key=SYSTEM,hive=NONE
Setting up the privileges database
Setting up idmap db
Setting up SAM db
Setting up sam.ldb partitions and settings
Setting up sam.ldb rootDSE
Pre-loading the Samba 4 and AD schema
partition_metadata: Migrating partition metadata: open of metadata.tdb gave: (null)
A Kerberos configuration suitable for Samba AD has been generated at /var/lib/samba/private/krb5.conf
Provision OK for domain DN DC=ad,DC=maxidom,DC=ru
Starting replication
Schema-DN[CN=Schema,CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[402/1550] linked_values[0/0]
Schema-DN[CN=Schema,CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[804/1550] linked_values[0/0]
Schema-DN[CN=Schema,CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[1206/1550] linked_values[0/0]
Schema-DN[CN=Schema,CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[1550/1550] linked_values[0/0]
Analyze and apply schema objects
Partition[CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[402/1649] linked_values[0/1]
Partition[CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[804/1649] linked_values[0/1]
Partition[CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[1206/1649] linked_values[0/1]
Partition[CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[1608/1649] linked_values[0/2]
Partition[CN=Configuration,DC=ad,DC=maxidom,DC=ru] objects[1649/1649] linked_values[72/72]
Replicating critical objects from the base DN of the domain
Partition[DC=ad,DC=maxidom,DC=ru] objects[100/100] linked_values[29/29]
Partition[DC=ad,DC=maxidom,DC=ru] objects[503/515] linked_values[0/30]
Partition[DC=ad,DC=maxidom,DC=ru] objects[615/515] linked_values[39/39]
Done with always replicated NC (base, config, schema)
Replicating DC=DomainDnsZones,DC=ad,DC=maxidom,DC=ru
Partition[DC=DomainDnsZones,DC=ad,DC=maxidom,DC=ru] objects[52/52] linked_values[0/0]
Replicating DC=ForestDnsZones,DC=ad,DC=maxidom,DC=ru
Partition[DC=ForestDnsZones,DC=ad,DC=maxidom,DC=ru] objects[22/22] linked_values[0/0]
Exop on[CN=RID Manager$,CN=System,DC=ad,DC=maxidom,DC=ru] objects[3] linked_values[0]
Committing SAM database
Adding 1 remote DNS records for DC1-U1.ad.maxidom.ru
Adding DNS A record DC1-U1.ad.maxidom.ru for IPv4 IP: 10.84.210.13
Adding DNS CNAME record 99b29c3b-2b91-49da-bd80-a4d708980fc4._msdcs.ad.maxidom.ru for DC1-U1.ad.maxidom.ru
All other DNS records (like _ldap SRV records) will be created samba_dnsupdate on first startup
Replicating new DNS records in DC=DomainDnsZones,DC=ad,DC=maxidom,DC=ru
Partition[DC=DomainDnsZones,DC=ad,DC=maxidom,DC=ru] objects[3/3] linked_values[0/0]
Replicating new DNS records in DC=ForestDnsZones,DC=ad,DC=maxidom,DC=ru
Partition[DC=ForestDnsZones,DC=ad,DC=maxidom,DC=ru] objects[2/2] linked_values[0/0]
Sending DsReplicaUpdateRefs for all the replicated partitions
Setting isSynchronized and dsServiceName
Setting up secrets database
Joined domain AD (SID S-1-5-21-2578626550-2309488607-201102032) as a DC
================================================================

Comment 4 Alexey Vekshin 2018-03-16 08:44:25 UTC

btw re-join with 4.7.6 was not completely flawless too: now logs on (some) of peer DCs are full of 

================================================================
2018.03.16 11:42:11 ad-pdc-vm(daemon.err) samba[11693]: [2018/03/16 11:42:11.781336,  0, pid=11693, effective(0, 0), real(0, 0)] ../source4/librpc/rpc/dcerpc_util.c:737(dcerpc_pipe_auth_recv)
2018.03.16 11:42:11 ad-pdc-vm(daemon.err) samba[11693]:   Failed to bind to uuid e3514235-4b06-11d1-ab04-00c04fc2dcd2 for ncacn_ip_tcp:10.84.128.10[49152,seal,krb5,target_hostname=267d7874-33e9-4919-9e56-27accda5bde3._msdcs.ad.maxidom.ru,target_principal=GC/dc1-u1.ad.maxidom.ru/ad.maxidom.ru,abstract_syntax=e3514235-4b06-11d1-ab04-00c04fc2dcd2/0x00000004,localaddress=10.0.45.74] NT_STATUS_UNSUCCESSFUL
================================================================

with corresponding errors on re-joined DC
================================================================
2018.03.16 13:42:11 dc1-u1-vm(daemon.warning) samba[30789]:   GSS server Update(krb5)(1) Update failed:  Miscellaneous failure (see text): Decrypt integrity check failed for checksum type hmac-sha1-96-aes256, key type aes256-cts-hmac-sha1-96
2018.03.16 13:42:11 dc1-u1-vm(daemon.warning) samba[30789]: [2018/03/16 13:42:11.760777,  1, pid=30789, effective(0, 0), real(0, 0)] ../source4/auth/gensec/gensec_gssapi.c:790(gensec_gssapi_update_internal)
================================================================

maybe related to https://forge.univention.org/bugzilla/show_bug.cgi?id=37358
or incomplete cleanup on demote

Comment 5 Alexey Vekshin 2018-06-07 14:07:01 UTC

works OK with 4.8.2