Bug 10991 - after classicupgrade: winbindd crash with "Failed to fetch our own, local AD domain join password for winbindd's internal use"
after classicupgrade: winbindd crash with "Failed to fetch our own, local AD ...
Status: RESOLVED FIXED
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Winbind
4.2.0rc2
x64 Linux
: P5 regression
: 4.3
Assigned To: Karolin Seeger
Samba QA Contact
:
: 10992 (view as bug list)
Depends on:
Blocks: 10695
  Show dependency treegraph
 
Reported: 2014-12-07 19:21 UTC by nrevel
Modified: 2015-07-05 19:02 UTC (History)
6 users (show)

See Also:


Attachments
Have winbindd sync secrets.tdb with secrets.ldb at startup (4.2 version) (7.74 KB, patch)
2015-06-12 02:19 UTC, Andrew Bartlett
no flags Details
Have winbindd sync secrets.tdb with secrets.ldb at startup (master version) (7.71 KB, patch)
2015-06-12 02:21 UTC, Andrew Bartlett
no flags Details
Have winbindd sync secrets.tdb with secrets.ldb at startup (4.2 version) (13.87 KB, patch)
2015-06-22 04:26 UTC, Andrew Bartlett
jra: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description nrevel 2014-12-07 19:21:34 UTC
My configuration is
DC1: (first DC upgraded with calssicupgrade to AD with 4.1.13) now under 4.2.0rc2
DC2: joined and promoted as a DC with 4.1.13 now under 4.2.0rc2

DC1 and DC2 use BIND9_DLZ dns backend
DC2 works well

with no modification in smb.conf, winbindd cause samba to crash with this lines in log:
/usr/sbin/winbindd: winbindd version 4.2.0rc2-18.1-3327-SUSE-oS13.1-x86_64 started.
/usr/sbin/winbindd: Copyright Andrew Tridgell and the Samba Team 1992-2014
/usr/sbin/winbindd: debug_lookup_classname(winbindd): Unknown class
/usr/sbin/winbindd: Maximum core file size limits now 16777216(soft) -1(hard)
/usr/sbin/winbindd: Registered MSG_REQ_POOL_USAGE
/usr/sbin/winbindd: Registered MSG_REQ_DMALLOC_MARK and LOG_CHANGED
/usr/sbin/winbindd: lp_load_ex: refreshing parameters
/usr/sbin/winbindd: Initialising global parameters
/usr/sbin/winbindd: rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
/usr/sbin/winbindd: Processing section "[global]"
/usr/sbin/winbindd: added interface enp2s0 ip=192.168.2.8 bcast=192.168.2.255 netmask=255.255.255.0
/usr/sbin/winbindd: added interface enp2s0 ip=192.168.2.8 bcast=192.168.2.255 netmask=255.255.255.0
/usr/sbin/winbindd: initialize_winbindd_cache: clearing cache and re-creating with version number 2
/usr/sbin/winbindd: Added domain BUILTIN (null) S-1-5-32
/usr/sbin/winbindd: Added domain XXXXXX XXXXXX.windows S-1-5-21-9999999999-9999999999-999999999
/usr/sbin/winbindd: Failed to fetch our own, local AD domain join password for winbindd's internal use
/usr/sbin/winbindd: unable to initialize domain list

This is my smb.conf
# Global parameters
[global]
        log level = 3 winbindd:255
        workgroup = XXXXXX 
        realm = XXXXXX.windows
        netbios name = DC1
        server role = active directory domain controller
        idmap_ldb:use rfc2307 = yes
        #server services = -dns +winbind -winbindd
        server services = -dns
        tls enabled = yes
        tls keyfile  = tls/DC1.key
        tls certfile = tls/DC2.pem
        tls cafile   = tls/CA.pem
        dsdb:schema update allowed = true
[netlogon]
        path = /var/lib/samba/sysvol/XXXXXX.windows/scripts
        read only = No

[sysvol]
        path = /var/lib/samba/sysvol
        read only = No

for having samba working I need to change the winbind in smb.conf:
server services = -dns +winbind -winbindd
Comment 1 Stefan Metzmacher 2014-12-08 09:08:20 UTC
Andrew, can you please have a look at this?
Comment 2 Stefan Metzmacher 2014-12-08 09:09:24 UTC
*** Bug 10992 has been marked as a duplicate of this bug. ***
Comment 3 Stefan Metzmacher 2014-12-18 08:05:32 UTC
This could be related to "idmap_ldb:use rfc2307 = yes" or a bug in the classic upgrade.

I have a domain with that was provisioned with 4.0.x, upgraded to 4.1
and running current master now. This works fine, I'll test with v4-2-test
but I guess this would also work...
Comment 4 Stefan Metzmacher 2015-01-26 21:24:04 UTC
I can't reproduce this with v4-2-test => no blocker of 4.2.0.
Comment 5 tsml412101 2015-03-23 14:54:55 UTC
I have run into this bug. Our domain was provisioned by classicupgrade going from 3.6.x to 4.0.0. We've since upgraded through every 4.0.x and 4.1.x version. I am currently trying to upgrade from 4.1.16 to 4.2.0.

I decided to restore from the 4.1.16 backup that I took before the upgrade, however I can setup a VM for testing.
Comment 6 and.win82 2015-03-26 09:39:24 UTC
I had the same problem when upgrading to 4.2.0. The domain has been provisioned by using classicupgrade going from a 3.6.x NT4-style domain to an 4.1.0 AD domain. I had to revert back to 4.1.17 to keep our production environment going.
Comment 7 Kelvin Yip 2015-04-09 03:50:38 UTC
I also have this problem after classicupgrade, will this related to secrets.tdb?
The following secrets.tdb is dump from classicupgrade(some are mask):
[root@linux01 kelvinyip]# tdbdump /usr/local/samba/private/secrets.tdb
{
key(23) = "SECRETS/PROTECT/IDS/ICS"
data(5) = "TRUE\00"
}
{
key(19) = "SECRETS/DOMGUID/ICS"
data(16) = ",X\XX\XX\XX\XXXX\XX\XXXX\0D\81\97\F0"
}
{
key(15) = "SECRETS/SID/ICS"
data(68) = "\01\04\00\00\00\00\00\05\15\00\00\00\E9\F1-\XX\XX\XX\XX\XX\XX\XX\XX\XX\00\00\00\00\00\00\00\00\00\00\00\00\00\                                                        00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00"
}

The following secrets.tdb is dump from new dc promption:
[root@linux Samba-Migrate]# tdbdump /usr/local/samba/private/secrets.tdb
{
key(39) = "SECRETS/SALTING_PRINCIPAL/DES/TEST1.LOC"
data(31) = "host/linux.test1.loc@TEST1.LOC\00"
}
{
key(38) = "SECRETS/MACHINE_LAST_CHANGE_TIME/TEST1"
data(4) = "O\F0%U"
}
{
key(38) = "SECRETS/MACHINE_SEC_CHANNEL_TYPE/TEST1"
data(4) = "\06\00\00\00"
}
{
key(25) = "SECRETS/PROTECT/IDS/TEST1"
data(5) = "TRUE\00"
}
{
key(30) = "SECRETS/MACHINE_PASSWORD/TEST1"
data(181) = "+d>!bxiFH=hInDPC.]<o:;$f~L%MAF,oB[KPnm(nRJxJ1?K8),7Db+Mh<y5q:MiSF@[7kB8;Bkd3au6J:BDG9I8CT,G%;>_jSE>HzMcU.&bhab,E.NQc78=BrIRdBF?LNjJ                                  VTlHIBVF&fDshQ#=PlDzK;ZgVdQqMYr?nvFGT$e_%TVPDB0gl\00"
}
{
key(21) = "SECRETS/DOMGUID/TEST1"
data(16) = "\8D\86]rH\E0wI\91\82\81\9C\92\07\93\9D"
}
{
key(17) = "SECRETS/SID/TEST1"
data(68) = "\01\04\00\00\00\00\00\05\15\00\00\00\E62\E67,\85^\93\92\A6\F5\F7\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\0                                  0\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00"
}
Comment 8 Kelvin Yip 2015-04-09 05:40:27 UTC
Seems my guess is correct, Samba4.2 can start now. I tried in my testing environment, and will try in production later.
I tried to dump the password through: 
ldbsearch --url=/usr/local/samba/private/secrets.ldb
Copy the "secrets" from the screen.

cp -rfp /usr/local/samba/private/secrets.tdb /usr/local/samba/private/secrets2.tdb 
#Execute the tdbtool to insert the record
tdbtool
>open /usr/local/samba/private/secrets.tdb
>insert SECRETS/SALTING_PRINCIPAL/DES/ICSHK.LOCAL host/linux01.icshk.local@ICSHK.LOCAL\00
>insert SECRETS/MACHINE_LAST_CHANGE_TIME/ICS O\F0%U
>insert SECRETS/MACHINE_SEC_CHANNEL_TYPE/ICS \06\00\00\00
>insert SECRETS/MACHINE_PASSWORD/ICS #Y3Y:@[>$j)Q.6Q<9P6)3=UH5gRgF_IRfO4J&c&)d(DV)cO:Z#[iMLK%>CdILs+i?W5(KKWe-=c0@?5ShxU:h6PFK;[ho5-]rDPsJp(79LH.b?CZZ<K3Ybdh-)(g+&:&GFbdABJa+pp,OgC;AanrPkie]9IR+QfWB1,jvK,MeC!wp3D-FkyqEXGOM:jzSaYq-;)\00
>quit

Modify the parameters above if you want to give a try.
Comment 9 Andrew Bartlett 2015-04-13 21:16:47 UTC
OK, this is starting to make more sense.  

Can you try running './source4/scripting/devel/chgtdcpass'

That should re-generate the passwords, and store them in the correct place. 

This looks like it is fixed in master, but in 4.2 we don't read the secrets.ldb to find the password during the upgrade case.
Comment 10 Andrew Bartlett 2015-04-22 08:53:46 UTC
The thing I find most confusing is that secrets.ldb should be writing out entries into secrets.tdb due to the secrets_tdb_sync module.  We need to know why that didn't execute in the classicupgrade.
Comment 11 Andrew Bartlett 2015-05-01 02:53:13 UTC
(In reply to Andrew Bartlett from comment #10)
The issue here is the secrets.tdb that is held open when we do the self join in the classicupgrade script is the OLD secrets.tdb file.

That means the DC self join password is placed in the secrets.tdb file in the Samba3 'source' directory. 

We need to either have the code from master to allow the newer of secrets.tdb or secrets.ldb to be used, or reset the secrets.tdb context held open in a static variable in secrets.c.

That could be done in the setup_self_join() code, but I don't like the side-effects.

Or we could rewrite secrets_store_machine_pw_sync() to take a db_ctx, which we would open in the secrets_tdb_sync module, but that would mean reworking the secrets_*() routines to take one, or not using those helper routines.

Andrew Bartlett
Comment 12 Andrew Bartlett 2015-05-01 09:37:05 UTC
(In reply to Andrew Bartlett from comment #11)
The simple fix for this is probably to:

Reset the open secrets.tdb in the script, and run chgtdcpass internally in the classicupgrade, essentially re-running the self join.
Comment 13 Luca Olivetti 2015-05-01 14:40:42 UTC
running chgtdcpass solved the problem in my 2 test cases:

1) classicupgrade to 4.1.17 then upgraded to 4.2.1
2) classicupgrade to 4.2.1
Comment 14 Andrew Bartlett 2015-06-12 02:19:39 UTC
Created attachment 11146 [details]
Have winbindd sync secrets.tdb with secrets.ldb at startup (4.2 version)

I'll update this patch with the master git commits when it is in master.  The master patch is slightly different.
Comment 15 Andrew Bartlett 2015-06-12 02:21:47 UTC
Created attachment 11147 [details]
Have winbindd sync secrets.tdb with secrets.ldb at startup (master version)

Patch for master, currently in a trial autobuild.
Comment 16 Antonio Rodríguez 2015-06-16 18:41:31 UTC
Hello 
Sorry forma mi english, i had this problem bit y dont know how lauch a chgtdcpass.
Thanks
Comment 17 Andrew Bartlett 2015-06-22 04:24:45 UTC
Comment on attachment 11147 [details]
Have winbindd sync secrets.tdb with secrets.ldb at startup (master version)

And improved version of this patch is now in master.
Comment 18 Andrew Bartlett 2015-06-22 04:26:20 UTC
Created attachment 11182 [details]
Have winbindd sync secrets.tdb with secrets.ldb at startup (4.2 version)

This patch backported from master (cherry-pick not possible due to test env rename)
Comment 19 Jeremy Allison 2015-06-22 20:57:20 UTC
Comment on attachment 11182 [details]
Have winbindd sync secrets.tdb with secrets.ldb at startup (4.2 version)

LGTM ( a few whitespace errors, but that shouldn't hurt applying, but please fix those in future Andrew).
Comment 20 Jeremy Allison 2015-06-22 20:57:55 UTC
Reassigning to Karolin for inclusion in 4.2.next.
Comment 21 Karolin Seeger 2015-06-29 19:59:17 UTC
(In reply to Jeremy Allison from comment #20)
Pushed to autobuild-v4-2-test.
Comment 22 Karolin Seeger 2015-07-05 19:02:15 UTC
Pushed to v4-2-test.
Closing out bug report.

Thanks!