Bug 12262 - net ads testjoin and smb access fails after winbindd changed the trust password
Summary: net ads testjoin and smb access fails after winbindd changed the trust password
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.1 and newer
Classification: Unclassified
Component: Winbind (show other bugs)
Version: 4.4.5
Hardware: x64 Linux
: P5 normal (vote)
Target Milestone: ---
Assignee: Karolin Seeger
QA Contact: Samba QA Contact
URL:
Keywords:
: 12465 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-09-14 11:38 UTC by Stefan Metzmacher
Modified: 2017-03-02 08:01 UTC (History)
6 users (show)

See Also:


Attachments
Reproducable password value... (3.18 KB, text/plain)
2017-01-18 15:47 UTC, Stefan Metzmacher
no flags Details
Work in progress patches for master (48.94 KB, patch)
2017-02-16 23:19 UTC, Stefan Metzmacher
no flags Details
Patches for v4-6-test (53.42 KB, patch)
2017-02-22 21:07 UTC, Stefan Metzmacher
slow: review+
Details
Patches for v4-5-test (53.62 KB, patch)
2017-02-22 21:08 UTC, Stefan Metzmacher
slow: review+
Details
Patches for v4-4-test (source3 only)(also applies to v4-3) (30.08 KB, patch)
2017-02-22 21:09 UTC, Stefan Metzmacher
slow: review+
Details
Minimal revert for v4-3-stable/v4-3-test (2.03 KB, patch)
2017-02-22 21:15 UTC, Stefan Metzmacher
slow: review+
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Metzmacher 2016-09-14 11:38:47 UTC
On ubuntu trusty x86_64 (with Version 4.4.5-SerNet-Ubuntu-16.trusty)

The domain join is partly broken by the weekly machine password
change.

wbinfo --check-secret
and
wbinfo --ping-dc

work fine.

I guess 'net rpc testjoin' would also work, but I didn't check that.

The problem is that any kerberos related authentication is broken
by the long random password.

net ads testjoin, "net ads search -P  '(name=administrator)'
and other commands fails.

ads_sasl_spnego_gensec_bind(KRB5) failed with: An internal error occurred., calling kinit

kerberos_kinit_password: as FS$@EXAMPLE.COM using [MEMORY:net_ads] as ccache and config [/var/cache/samba/smb_krb5/krb5.conf.EXAMPLE]
kerberos_kinit_password FS$@EXAMPLE.COM failed: Preauthentication failed
Join to domain is not valid: Logon failure
return code = -1

And SMB clients can't connect anymore:

[2016/09/13 12:44:44.357493,  1, pid=1234] ../source3/librpc/crypto/gse.c:497(gse_get_server_auth_token)
  gss_accept_sec_context failed with [ Miscellaneous failure (see text): Failed to find cifs/fs.example.com@EXAMPLE.COM(kvno 10) in keytab MEMORY:cifs_srv_keytab (aes256-cts-hmac-sha1-96)]
[2016/09/13 12:44:44.357592,  1, pid=1234] ../auth/gensec/spnego.c:541(gensec_spnego_parse_negTokenInit)
  SPNEGO(gse_krb5) NEG_TOKEN_INIT failed: NT_STATUS_LOGON_FAILURE

A server in the same domain using debian jessie works fine.

Both servers have msDs-SupportedEncryptionTypes: 31

A 'net ads join', which sets a shorter and less random password
fixes the situation again.
Comment 1 Stefan Metzmacher 2017-01-18 15:47:08 UTC
Created attachment 12843 [details]
Reproducable password value...

I was finally able to reproduce this. It turns out that
encode_pw_buffer() and extract_pw_from_buffer() are not
symmetric and (at least) one of them is wrong for random utf8 strings.
Comment 2 Ralph Böhme 2017-01-18 15:49:42 UTC
Comment on attachment 12843 [details]
Reproducable password value...

Wrong patch?
Comment 3 Ralph Böhme 2017-01-18 15:50:37 UTC
Hooray metze! :)
Comment 4 Stefan Metzmacher 2017-01-18 18:01:49 UTC
Comment on attachment 12843 [details]
Reproducable password value...

Yes wrong patch...

The password I'm testing with is 

key(39) = "SECRETS/MACHINE_PASSWORD.PREV/W4EDOM-L4"
data(43) = "\E8\9A\AA\E4\A9\99\E9\81\AF\E3\9A\AB\E0\B2\89\EE\B9\8D\EB\81\86\E4\8F\9C\E7\95\85\E9\89\85\E2\88\85\E2\8F\85\E8\99\9A\E9\BC\84\00"
Comment 5 Stefan Metzmacher 2017-01-31 08:42:17 UTC
(In reply to Stefan Metzmacher from comment #4)

It turns out that I only reproduced it by machine_password[0]++;
in the code :-(
Comment 6 Alban Rodriguez 2017-02-03 13:20:52 UTC
Hi,

Is any patch has been merged into a current release please ?
I did not see anything related in the change logs for release 4.5.5 for instance.

Now that branch 4.6 is out, I guess it won't make its way into the 4.4 branch ?

Thank you
Alban


>https://bugzilla.samba.org/show_bug.cgi?id=12465
>
>Stefan Metzmacher <metze@samba.org> changed:
>
>          What    |Removed                     |Added
>----------------------------------------------------------------------------
>          See Also|                            |https://bugzilla.samba.org/
>                  |                            |show_bug.cgi?id=12262
>
>-- 
>You are receiving this mail because:
>You reported the bug.
>
Comment 7 Stefan Metzmacher 2017-02-03 14:47:32 UTC
(In reply to Alban Rodriguez from comment #6)

I haven't found the root cause of the problem yet
and it seems it doesn't happen each time.

So there's no patch.

4.4 won't get it once 4.6.0 (final) is released.
I but guess a 4.5 patch would still apply to 4.4
Comment 8 Stefan Metzmacher 2017-02-03 14:49:07 UTC
(In reply to Stefan Metzmacher from comment #7)

If someone can reproduce this at will let me know.
On a testmachine it would be perfect...
Comment 9 Stefan Metzmacher 2017-02-10 13:41:52 UTC
(In reply to Stefan Metzmacher from comment #8)

I was finally able to reproduce this, using 'dos charset = CP850'
and 'unix charset = ISO8859-1'.

I guess the problem is the conversion from

UTF16MUNGED => UTF8 when generating the password,
which is a value no longer valid in the configured 'unix charset'.
Comment 10 Ralph Böhme 2017-02-10 13:50:06 UTC
go, metze, go! :)))
Comment 11 Stefan Metzmacher 2017-02-10 14:51:34 UTC
The member sends this:

[2017/02/10 15:47:57.154969,  0, pid=1474, class=passdb] ../source3/passdb/machine_account_secrets.c:446(secrets_store_machine_password)
  ../source3/passdb/machine_account_secrets.c:446:secrets_store_machine_password: pw.length[43]
[2017/02/10 15:47:57.155111,  0, pid=1474] ../lib/util/util.c:559(dump_data)
  [0000] E5 A6 89 E8 8F B4 ED 81   8A ED 8B BB E6 95 96 E9   ........ ........
  [0010] B9 82 E9 9A B5 EB A5 A3   EA 9B B9 E8 9C B3 E9 A5   ........ ........
  [0020] B2 E7 9D 9F E2 A8 9A E0   BA 93 00                  ........ ...
[2017/02/10 15:47:57.236499,  0, pid=1474] ../source3/libsmb/trusts_util.c:296(trust_pw_change)
  2017/02/10 15:47:57 : trust_pw_change(W4EDOM-L4): Changed password locally
[2017/02/10 15:47:57.236640,  0, pid=1474] ../libcli/auth/smbencrypt.c:715(encode_pw_buffer_debug)
  ../libcli/auth/smbencrypt.c:715:encode_pw_buffer_debug: pw_buffer[84]
  upper[E500A6008900E8008F00B400ED0081008A00ED008B00BB00E60095009600E900B9008200E9009A00B500EB00A500A300EA009B00B900E8009C00B300E900A500B200E7009D009F00E200A8009A00E000BA009300]
  lower[e500a6008900e8008f00b400ed0081008a00ed008b00bb00e60095009600e900b9008200e9009a00b500eb00a500a300ea009b00b900e8009c00b300e900a500b200e7009d009f00e200a8009a00e000ba009300]

And the following arrives on the server:

[2017/02/10 15:47:57.242101,  0, pid=2346] ../source4/dsdb/samdb/ldb_modules/password_hash.c:2177(setup_given_passwords)
  ../source4/dsdb/samdb/ldb_modules/password_hash.c:2177:setup_given_passwords: utf16[84]
  upper[E500A6008900E8008F00B400ED0081008A00ED008B00BB00E60095009600E900B9008200E9009A00B500EB00A500A300EA009B00B900E8009C00B300E900A500B200E7009D009F00E200A8009A00E000BA009300]
  lower[e500a6008900e8008f00b400ed0081008a00ed008b00bb00e60095009600e900b9008200e9009a00b500eb00a500a300ea009b00b900e8009c00b300e900a500b200e7009d009f00e200a8009a00e000ba009300]
[2017/02/10 15:47:57.242188,  0, pid=2346] ../source4/dsdb/samdb/ldb_modules/password_hash.c:2181(setup_given_passwords)
  ../source4/dsdb/samdb/ldb_modules/password_hash.c:2181:setup_given_passwords: utf8[84]
[2017/02/10 15:47:57.242239,  0, pid=2346] ../lib/util/util.c:555(dump_data)
  [0000] C3 A5 C2 A6 C2 89 C3 A8   C2 8F C2 B4 C3 AD C2 81   ........ ........
  [0010] C2 8A C3 AD C2 8B C2 BB   C3 A6 C2 95 C2 96 C3 A9   ........ ........
  [0020] C2 B9 C2 82 C3 A9 C2 9A   C2 B5 C3 AB C2 A5 C2 A3   ........ ........
  [0030] C3 AA C2 9B C2 B9 C3 A8   C2 9C C2 B3 C3 A9 C2 A5   ........ ........
  [0040] C2 B2 C3 A7 C2 9D C2 9F   C3 A2 C2 A8 C2 9A C3 A0   ........ ........
  [0050] C2 BA C2 93                                        .... 
[2017/02/10 15:47:57.242397,  0, pid=2346] ../source4/dsdb/samdb/ldb_modules/password_hash.c:2186(setup_given_passwords)
  ../source4/dsdb/samdb/ldb_modules/password_hash.c:2186:setup_given_passwords: nthash
[2017/02/10 15:47:57.242428,  0, pid=2346] ../lib/util/util.c:555(dump_data)
  [0000] 37 B2 C8 FF 75 FC 4A 7A   6D C3 D7 65 EE 55 8F 79   7...u.Jz m..e.U.y
[2017/02/10 15:47:57.242589,  0, pid=2346] ../lib/krb5_wrap/krb5_samba.c:255(smb_krb5_create_key_from_string_debug)
  ../lib/krb5_wrap/krb5_samba.c:255:smb_krb5_create_key_from_string_debug: enctype[0x12] password->length[84] strlen(84) salt[W4EDOM-L4.BASEhostub1404-162.w4edom-l4.base]
[2017/02/10 15:47:57.242680,  0, pid=2346] ../lib/util/util.c:555(dump_data)
  [0000] C3 A5 C2 A6 C2 89 C3 A8   C2 8F C2 B4 C3 AD C2 81   ........ ........
  [0010] C2 8A C3 AD C2 8B C2 BB   C3 A6 C2 95 C2 96 C3 A9   ........ ........
  [0020] C2 B9 C2 82 C3 A9 C2 9A   C2 B5 C3 AB C2 A5 C2 A3   ........ ........
  [0030] C3 AA C2 9B C2 B9 C3 A8   C2 9C C2 B3 C3 A9 C2 A5   ........ ........
  [0040] C2 B2 C3 A7 C2 9D C2 9F   C3 A2 C2 A8 C2 9A C3 A0   ........ ........
  [0050] C2 BA C2 93                                        .... 
[2017/02/10 15:47:57.317104,  0, pid=2346] ../lib/krb5_wrap/krb5_samba.c:276(smb_krb5_create_key_from_string_debug)
  ../lib/krb5_wrap/krb5_samba.c:276:smb_krb5_create_key_from_string_debug: ret[0] keytype[0x12] key->length[32]
[2017/02/10 15:47:57.317173,  0, pid=2346] ../lib/util/util.c:555(dump_data)
  [0000] 49 F2 B8 8B A8 C2 6D 79   98 96 C3 FE 21 2B 90 68   I.....my ....!+.h
  [0010] CF 64 AF 71 13 0F 12 0B   20 28 63 24 3D DF 22 5E   .d.q....  (c$=."^
Comment 12 Stefan Metzmacher 2017-02-10 15:19:27 UTC
(In reply to Stefan Metzmacher from comment #11)

net rpc testjoin and wbinfo --ping-dc still work
as we calculate the same NTHASH in our code.

But kerberos is completely broken, even with
arcfour-hmac-md5, at least with heimdal as ARCFOUR_string_to_key()
doesn't generate the same NTHASH. I guess
because of wind_utf8ucs2[_length]() and differences between UCS2 and UTF-16LE.
Comment 13 Stefan Metzmacher 2017-02-16 11:54:38 UTC
(In reply to Stefan Metzmacher from comment #12)

We may also have problem is strange random cases where
generate_random_buffer() generates a valid 
high surrogate (0xD800 - 0xDBFF) followed by
a valid low surrogate (0xDC00 - 0xDFFF).

This results in a codepoint > 0xFFFF.

It turns out that both MIT krb5 and Heimdal only
support codepoints up to 0xFFFF while generating
the arcfour-hmac-md5 key (the NTHASH).

As we ignore failures from create_kerberos_key_from_string()
in fill_keytab_from_password() we may generate an in memory keytab
without the arcfour-hmac-md5 key. If the account on the DC
doesn't specify msDs-SupportedEncryptionTypes for aes usage.
(this typically the case when the join happened with on older Samba version (< 4.2)).

I'll upload patches hopefully soon.
Comment 14 Andrew Bartlett 2017-02-16 18:07:56 UTC
(In reply to Stefan Metzmacher from comment #13)
Thanks.  I'm pretty sure this latest point is a flapping test we see occasionally.
Comment 15 Stefan Metzmacher 2017-02-16 23:17:14 UTC
(In reply to Andrew Bartlett from comment #14)

Which one?
Comment 16 Stefan Metzmacher 2017-02-16 23:19:04 UTC
Created attachment 12951 [details]
Work in progress patches for master

I'm currently trying to get this passing autobuild,
the last time I got failures in the python code with encoding mismatches...
Comment 17 Andrew Bartlett 2017-02-16 23:37:13 UTC
(In reply to Stefan Metzmacher from comment #15)
I've seen autobuild failing with string2key errors.  I'll see if I can find more specific details for you.
Comment 18 Stefan Metzmacher 2017-02-22 19:34:39 UTC
*** Bug 12465 has been marked as a duplicate of this bug. ***
Comment 19 Stefan Metzmacher 2017-02-22 21:07:29 UTC
Created attachment 12970 [details]
Patches for v4-6-test
Comment 20 Stefan Metzmacher 2017-02-22 21:08:15 UTC
Created attachment 12971 [details]
Patches for v4-5-test
Comment 21 Stefan Metzmacher 2017-02-22 21:09:11 UTC
Created attachment 12972 [details]
Patches for v4-4-test (source3 only)(also applies to v4-3)
Comment 22 Stefan Metzmacher 2017-02-22 21:15:19 UTC
Created attachment 12973 [details]
Minimal revert for v4-3-stable/v4-3-test
Comment 23 Ralph Böhme 2017-02-23 06:28:16 UTC
Thanks for all the work metze!

Reassigning to Karolin for inclusion in 4.4, 4.5 and 4.6.
Comment 24 Karolin Seeger 2017-02-24 08:26:17 UTC
(In reply to Ralph Böhme from comment #23)
Pushed to autobuild-v4-{6,5,4}-test.
v4-4-test is closed, but this is one is critical, so I pushed it anyway.
Comment 25 Karolin Seeger 2017-03-02 08:01:47 UTC
(In reply to Karolin Seeger from comment #24)
Pushed to all branches.
Closing out bug report.

Thanks!